2024-09-16T04:26:36.397 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T04:26:36.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:26:36.506 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025 branch: squid description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906025' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfoMci92qZ165G6JxDwvmtdC/wRtQ96bicnhUZm7FeyEx/UdlIHSyYx5u4/pewC+Kmn8OxtrnreeJ01cZbxph4= smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMLWCwfak+9LK8UE4aubQwW/RuPFQviJl5MRUFwJFPs8PzRBIgRcUUyn1Rs4sbQk8X6bMc7TU8NPkUtQHMqQYVA= smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDi7RBJQj/3kKjeOdiBKn75cafN5UKG45TvOVRfbuQLJRw4JTYJrtd8Hy8Dpo9kOCj09HV/u8fk37woXZGZ5AjQ= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFgy3D4vw9464Msud1VfhXL6DNpoVJfPqQzUM9EWTBnKmO5X3d4XDp2GubM9lloafDMbqL0fJGOfjqzFEUvZlzw= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T04:26:36.506 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T04:26:36.507 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T04:26:36.507 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T04:26:36.509 INFO:teuthology.task.internal:Checking packages... 2024-09-16T04:26:36.530 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T04:26:36.530 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:26:36.530 INFO:teuthology.packaging:ref: None 2024-09-16T04:26:36.530 INFO:teuthology.packaging:tag: None 2024-09-16T04:26:36.530 INFO:teuthology.packaging:branch: squid 2024-09-16T04:26:36.530 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:26:36.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-16T04:26:36.736 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:26:36.738 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T04:26:36.744 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T04:26:36.744 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T04:26:36.762 INFO:teuthology.task.internal:Saving configuration 2024-09-16T04:26:36.779 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T04:26:36.792 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T04:26:36.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:21:00.935818', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKa6QeXUbPvQJjezbSIrKA7soulhQtgswQEmTXm1dZYj'} 2024-09-16T04:26:36.839 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:21:00.937664', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH45UVOLnckj7c74k9KDNvxmVKWMDV3u0bW1NrmbxoGX'} 2024-09-16T04:26:36.865 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:21:00.940226', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEW1Rd+yO9jhn05aYF8jitjLGoSDTbnYYkvnK8nfrQgf'} 2024-09-16T04:26:36.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:21:00.938938', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB4pNfUjRv9kPCACcpb94mvg9z5F/kk66vivU5qXv0A4'} 2024-09-16T04:26:36.888 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T04:26:36.899 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-16T04:26:36.900 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-16T04:26:36.900 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-16T04:26:36.900 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-16T04:26:36.900 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T04:26:37.015 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0c34566e60>, signals=[15]) 2024-09-16T04:26:37.015 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T04:26:37.025 INFO:teuthology.task.internal:Opening connections... 2024-09-16T04:26:37.025 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:26:37.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:26:37.109 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:26:37.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:26:37.186 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:26:37.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:26:37.264 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:26:37.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:26:37.345 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T04:26:37.353 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-09-16T04:26:37.359 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-09-16T04:26:37.359 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="Ubuntu" 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="22.04" 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_CODENAME=jammy 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:ID=ubuntu 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE=debian 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T04:26:37.406 INFO:teuthology.orchestra.run.smithi059.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T04:26:37.407 INFO:teuthology.orchestra.run.smithi059.stdout:UBUNTU_CODENAME=jammy 2024-09-16T04:26:37.407 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-09-16T04:26:37.429 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-09-16T04:26:37.435 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-09-16T04:26:37.436 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-09-16T04:26:37.482 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T04:26:37.482 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="Ubuntu" 2024-09-16T04:26:37.482 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="22.04" 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_CODENAME=jammy 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:ID=ubuntu 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE=debian 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T04:26:37.483 INFO:teuthology.orchestra.run.smithi139.stdout:UBUNTU_CODENAME=jammy 2024-09-16T04:26:37.483 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-09-16T04:26:37.505 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2024-09-16T04:26:37.510 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2024-09-16T04:26:37.511 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-09-16T04:26:37.557 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T04:26:37.557 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="Ubuntu" 2024-09-16T04:26:37.557 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="22.04" 2024-09-16T04:26:37.557 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T04:26:37.557 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_CODENAME=jammy 2024-09-16T04:26:37.558 INFO:teuthology.orchestra.run.smithi144.stdout:ID=ubuntu 2024-09-16T04:26:37.558 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE=debian 2024-09-16T04:26:37.558 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T04:26:37.558 INFO:teuthology.orchestra.run.smithi144.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T04:26:37.558 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T04:26:37.558 INFO:teuthology.orchestra.run.smithi144.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T04:26:37.558 INFO:teuthology.orchestra.run.smithi144.stdout:UBUNTU_CODENAME=jammy 2024-09-16T04:26:37.558 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-09-16T04:26:37.579 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-09-16T04:26:37.585 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-09-16T04:26:37.585 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-09-16T04:26:37.631 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="Ubuntu" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="22.04" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_CODENAME=jammy 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:ID=ubuntu 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE=debian 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T04:26:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:UBUNTU_CODENAME=jammy 2024-09-16T04:26:37.633 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-09-16T04:26:37.655 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T04:26:37.664 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T04:26:37.670 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T04:26:37.670 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:26:37.672 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:26:37.675 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:26:37.678 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:26:37.683 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T04:26:37.689 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T04:26:37.689 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:26:37.718 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:26:37.721 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:26:37.726 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:26:37.733 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T04:26:37.835 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T04:26:37.910 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T04:26:37.910 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T04:26:37.910 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:26:37.911 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:26:37.911 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:26:37.912 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:26:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:26:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:26:37.918 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-09-16T04:26:37.918 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-16T04:26:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:26:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:26:37.920 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-16T04:26:37.921 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-09-16T04:26:37.964 INFO:teuthology.orchestra.run.smithi139.stdout:5.15.0-56-generic 2024-09-16T04:26:37.965 INFO:teuthology.task.kernel:Running kernel on smithi139: 5.15.0-56-generic 2024-09-16T04:26:37.965 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get clean 2024-09-16T04:26:37.970 INFO:teuthology.orchestra.run.smithi144.stdout:5.15.0-56-generic 2024-09-16T04:26:37.970 INFO:teuthology.task.kernel:Running kernel on smithi144: 5.15.0-56-generic 2024-09-16T04:26:37.971 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get clean 2024-09-16T04:26:37.972 INFO:teuthology.orchestra.run.smithi165.stdout:5.15.0-56-generic 2024-09-16T04:26:37.972 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.15.0-56-generic 2024-09-16T04:26:37.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2024-09-16T04:26:37.974 INFO:teuthology.orchestra.run.smithi059.stdout:5.15.0-56-generic 2024-09-16T04:26:37.974 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.15.0-56-generic 2024-09-16T04:26:37.974 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2024-09-16T04:26:38.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-16T04:26:38.094 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get update 2024-09-16T04:26:38.102 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-09-16T04:26:38.133 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get update 2024-09-16T04:26:38.246 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:26:38.246 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T04:26:38.246 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T04:26:38.275 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:26:38.284 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T04:26:38.315 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T04:26:38.355 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T04:26:38.357 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:26:38.357 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T04:26:38.367 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:26:38.368 INFO:teuthology.orchestra.run.smithi139.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T04:26:38.384 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T04:26:38.413 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T04:26:38.423 INFO:teuthology.orchestra.run.smithi139.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T04:26:38.496 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T04:26:38.503 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T04:26:38.568 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T04:26:38.585 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T04:26:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T04:26:38.604 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T04:26:38.607 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T04:26:38.610 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T04:26:38.611 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T04:26:38.611 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T04:26:38.612 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T04:26:38.613 INFO:teuthology.orchestra.run.smithi144.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T04:26:38.632 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T04:26:38.632 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T04:26:38.636 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T04:26:38.636 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T04:26:38.643 INFO:teuthology.orchestra.run.smithi144.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T04:26:38.643 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T04:26:38.647 INFO:teuthology.orchestra.run.smithi144.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T04:26:38.647 INFO:teuthology.orchestra.run.smithi144.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T04:26:38.648 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T04:26:38.649 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T04:26:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T04:26:38.654 INFO:teuthology.orchestra.run.smithi144.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T04:26:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T04:26:38.662 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T04:26:38.663 INFO:teuthology.orchestra.run.smithi144.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T04:26:38.665 INFO:teuthology.orchestra.run.smithi144.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T04:26:38.665 INFO:teuthology.orchestra.run.smithi144.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T04:26:38.686 INFO:teuthology.orchestra.run.smithi144.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T04:26:38.686 INFO:teuthology.orchestra.run.smithi144.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T04:26:38.686 INFO:teuthology.orchestra.run.smithi144.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T04:26:38.686 INFO:teuthology.orchestra.run.smithi144.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T04:26:38.687 INFO:teuthology.orchestra.run.smithi144.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T04:26:38.687 INFO:teuthology.orchestra.run.smithi144.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T04:26:38.693 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T04:26:38.693 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T04:26:38.696 INFO:teuthology.orchestra.run.smithi144.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T04:26:38.696 INFO:teuthology.orchestra.run.smithi144.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T04:26:38.705 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T04:26:38.712 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T04:26:38.713 INFO:teuthology.orchestra.run.smithi144.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T04:26:38.714 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T04:26:38.714 INFO:teuthology.orchestra.run.smithi144.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T04:26:38.714 INFO:teuthology.orchestra.run.smithi144.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T04:26:38.752 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T04:26:38.752 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T04:26:38.757 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T04:26:38.757 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T04:26:38.768 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T04:26:38.769 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T04:26:38.774 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T04:26:38.777 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T04:26:38.777 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T04:26:38.778 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T04:26:38.778 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T04:26:38.779 INFO:teuthology.orchestra.run.smithi139.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T04:26:38.785 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T04:26:38.788 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T04:26:38.789 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T04:26:38.789 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T04:26:38.790 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T04:26:38.795 INFO:teuthology.orchestra.run.smithi139.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T04:26:38.805 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T04:26:38.805 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T04:26:38.806 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T04:26:38.806 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T04:26:38.807 INFO:teuthology.orchestra.run.smithi059.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T04:26:38.837 INFO:teuthology.orchestra.run.smithi144.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T04:26:38.908 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T04:26:38.919 INFO:teuthology.orchestra.run.smithi139.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T04:26:39.089 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T04:26:39.094 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T04:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T04:26:39.237 INFO:teuthology.orchestra.run.smithi139.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T04:26:39.257 INFO:teuthology.orchestra.run.smithi144.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T04:26:39.261 INFO:teuthology.orchestra.run.smithi139.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T04:26:39.262 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T04:26:39.267 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T04:26:39.273 INFO:teuthology.orchestra.run.smithi144.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T04:26:39.276 INFO:teuthology.orchestra.run.smithi139.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T04:26:39.277 INFO:teuthology.orchestra.run.smithi139.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T04:26:39.279 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T04:26:39.279 INFO:teuthology.orchestra.run.smithi139.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T04:26:39.280 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T04:26:39.280 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T04:26:39.281 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T04:26:39.282 INFO:teuthology.orchestra.run.smithi144.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T04:26:39.282 INFO:teuthology.orchestra.run.smithi139.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T04:26:39.282 INFO:teuthology.orchestra.run.smithi144.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T04:26:39.289 INFO:teuthology.orchestra.run.smithi139.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T04:26:39.290 INFO:teuthology.orchestra.run.smithi139.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T04:26:39.292 INFO:teuthology.orchestra.run.smithi139.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T04:26:39.346 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T04:26:39.347 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T04:26:39.361 INFO:teuthology.orchestra.run.smithi139.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T04:26:39.376 INFO:teuthology.orchestra.run.smithi139.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T04:26:39.376 INFO:teuthology.orchestra.run.smithi139.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T04:26:39.378 INFO:teuthology.orchestra.run.smithi144.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T04:26:39.380 INFO:teuthology.orchestra.run.smithi144.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T04:26:39.381 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T04:26:39.381 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T04:26:39.381 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T04:26:39.381 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T04:26:39.389 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T04:26:39.389 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T04:26:39.390 INFO:teuthology.orchestra.run.smithi144.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T04:26:39.390 INFO:teuthology.orchestra.run.smithi144.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T04:26:39.401 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T04:26:39.406 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T04:26:39.413 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T04:26:39.414 INFO:teuthology.orchestra.run.smithi144.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T04:26:39.417 INFO:teuthology.orchestra.run.smithi139.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T04:26:39.417 INFO:teuthology.orchestra.run.smithi139.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T04:26:39.419 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T04:26:39.426 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T04:26:39.427 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T04:26:39.427 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T04:26:39.427 INFO:teuthology.orchestra.run.smithi139.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T04:26:39.427 INFO:teuthology.orchestra.run.smithi139.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T04:26:39.432 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T04:26:39.433 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T04:26:39.433 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T04:26:39.435 INFO:teuthology.orchestra.run.smithi144.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T04:26:39.439 INFO:teuthology.orchestra.run.smithi144.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T04:26:39.442 INFO:teuthology.orchestra.run.smithi144.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T04:26:39.443 INFO:teuthology.orchestra.run.smithi144.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T04:26:39.444 INFO:teuthology.orchestra.run.smithi139.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T04:26:39.448 INFO:teuthology.orchestra.run.smithi139.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T04:26:39.450 INFO:teuthology.orchestra.run.smithi139.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T04:26:39.450 INFO:teuthology.orchestra.run.smithi139.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T04:26:39.452 INFO:teuthology.orchestra.run.smithi139.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T04:26:39.452 INFO:teuthology.orchestra.run.smithi139.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T04:26:39.457 INFO:teuthology.orchestra.run.smithi139.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T04:26:39.460 INFO:teuthology.orchestra.run.smithi139.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T04:26:39.460 INFO:teuthology.orchestra.run.smithi139.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T04:26:39.461 INFO:teuthology.orchestra.run.smithi139.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T04:26:39.461 INFO:teuthology.orchestra.run.smithi139.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T04:26:39.486 INFO:teuthology.orchestra.run.smithi139.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T04:26:39.486 INFO:teuthology.orchestra.run.smithi139.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T04:26:39.487 INFO:teuthology.orchestra.run.smithi139.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T04:26:39.497 INFO:teuthology.orchestra.run.smithi165.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T04:26:39.497 INFO:teuthology.orchestra.run.smithi165.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T04:26:39.497 INFO:teuthology.orchestra.run.smithi165.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T04:26:39.499 INFO:teuthology.orchestra.run.smithi165.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T04:26:39.500 INFO:teuthology.orchestra.run.smithi165.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T04:26:39.507 INFO:teuthology.orchestra.run.smithi139.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T04:26:39.508 INFO:teuthology.orchestra.run.smithi165.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T04:26:39.508 INFO:teuthology.orchestra.run.smithi165.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T04:26:39.509 INFO:teuthology.orchestra.run.smithi165.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T04:26:39.513 INFO:teuthology.orchestra.run.smithi165.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T04:26:39.514 INFO:teuthology.orchestra.run.smithi165.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T04:26:39.523 INFO:teuthology.orchestra.run.smithi144.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T04:26:39.523 INFO:teuthology.orchestra.run.smithi144.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T04:26:39.534 INFO:teuthology.orchestra.run.smithi139.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T04:26:39.535 INFO:teuthology.orchestra.run.smithi139.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T04:26:39.535 INFO:teuthology.orchestra.run.smithi139.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T04:26:39.535 INFO:teuthology.orchestra.run.smithi139.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T04:26:39.536 INFO:teuthology.orchestra.run.smithi139.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T04:26:39.567 INFO:teuthology.orchestra.run.smithi139.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T04:26:39.589 INFO:teuthology.orchestra.run.smithi165.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T04:26:39.590 INFO:teuthology.orchestra.run.smithi165.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T04:26:45.008 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 16.0 MB in 2s (7,648 kB/s) 2024-09-16T04:26:45.342 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 16.0 MB in 2s (7,074 kB/s) 2024-09-16T04:26:45.471 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 16.0 MB in 3s (5,812 kB/s) 2024-09-16T04:26:45.526 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 16.0 MB in 3s (5,902 kB/s) 2024-09-16T04:26:46.598 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-16T04:26:46.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:26:46.679 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-16T04:26:46.835 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-09-16T04:26:46.854 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:26:46.888 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2024-09-16T04:26:46.889 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2024-09-16T04:26:46.921 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-09-16T04:26:46.978 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-16T04:26:46.983 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-16T04:26:46.996 DEBUG:teuthology.orchestra.run.smithi139:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:26:47.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:26:47.057 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-16T04:26:47.062 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-16T04:26:47.065 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:26:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:26:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:26:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:26:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T04:26:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: python2.7-minimal 2024-09-16T04:26:47.066 INFO:teuthology.orchestra.run.smithi144.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:26:47.067 INFO:teuthology.orchestra.run.smithi144.stdout:The following additional packages will be installed: 2024-09-16T04:26:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T04:26:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.069 INFO:teuthology.orchestra.run.smithi144.stdout:Suggested packages: 2024-09-16T04:26:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T04:26:47.069 INFO:teuthology.orchestra.run.smithi144.stdout:Recommended packages: 2024-09-16T04:26:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: thermald 2024-09-16T04:26:47.121 INFO:teuthology.orchestra.run.smithi144.stdout:The following NEW packages will be installed: 2024-09-16T04:26:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T04:26:47.122 INFO:teuthology.orchestra.run.smithi144.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.123 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages will be upgraded: 2024-09-16T04:26:47.123 INFO:teuthology.orchestra.run.smithi144.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T04:26:47.135 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-09-16T04:26:47.135 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-09-16T04:26:47.252 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-16T04:26:47.252 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-16T04:26:47.266 INFO:teuthology.orchestra.run.smithi139.stdout:Building dependency tree... 2024-09-16T04:26:47.267 INFO:teuthology.orchestra.run.smithi139.stdout:Reading state information... 2024-09-16T04:26:47.302 INFO:teuthology.orchestra.run.smithi144.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:26:47.302 INFO:teuthology.orchestra.run.smithi144.stdout:Need to get 113 MB of archives. 2024-09-16T04:26:47.302 INFO:teuthology.orchestra.run.smithi144.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T04:26:47.302 INFO:teuthology.orchestra.run.smithi144.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T04:26:47.329 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:26:47.330 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:26:47.330 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:26:47.330 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:26:47.330 INFO:teuthology.orchestra.run.smithi059.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T04:26:47.330 INFO:teuthology.orchestra.run.smithi059.stdout: python2.7-minimal 2024-09-16T04:26:47.331 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:26:47.332 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2024-09-16T04:26:47.332 INFO:teuthology.orchestra.run.smithi059.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.332 INFO:teuthology.orchestra.run.smithi059.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T04:26:47.333 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.334 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2024-09-16T04:26:47.334 INFO:teuthology.orchestra.run.smithi059.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T04:26:47.334 INFO:teuthology.orchestra.run.smithi059.stdout:Recommended packages: 2024-09-16T04:26:47.334 INFO:teuthology.orchestra.run.smithi059.stdout: thermald 2024-09-16T04:26:47.386 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2024-09-16T04:26:47.387 INFO:teuthology.orchestra.run.smithi059.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.387 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T04:26:47.387 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.388 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages will be upgraded: 2024-09-16T04:26:47.389 INFO:teuthology.orchestra.run.smithi059.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T04:26:47.415 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:26:47.415 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:26:47.416 INFO:teuthology.orchestra.run.smithi165.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:26:47.416 INFO:teuthology.orchestra.run.smithi165.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:26:47.416 INFO:teuthology.orchestra.run.smithi165.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T04:26:47.416 INFO:teuthology.orchestra.run.smithi165.stdout: python2.7-minimal 2024-09-16T04:26:47.416 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:26:47.418 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2024-09-16T04:26:47.418 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.418 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T04:26:47.418 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.419 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2024-09-16T04:26:47.419 INFO:teuthology.orchestra.run.smithi165.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T04:26:47.420 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2024-09-16T04:26:47.420 INFO:teuthology.orchestra.run.smithi165.stdout: thermald 2024-09-16T04:26:47.440 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:26:47.440 INFO:teuthology.orchestra.run.smithi139.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:26:47.440 INFO:teuthology.orchestra.run.smithi139.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:26:47.441 INFO:teuthology.orchestra.run.smithi139.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:26:47.441 INFO:teuthology.orchestra.run.smithi139.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T04:26:47.441 INFO:teuthology.orchestra.run.smithi139.stdout: python2.7-minimal 2024-09-16T04:26:47.441 INFO:teuthology.orchestra.run.smithi139.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:26:47.443 INFO:teuthology.orchestra.run.smithi139.stdout:The following additional packages will be installed: 2024-09-16T04:26:47.443 INFO:teuthology.orchestra.run.smithi139.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.443 INFO:teuthology.orchestra.run.smithi139.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T04:26:47.443 INFO:teuthology.orchestra.run.smithi139.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.444 INFO:teuthology.orchestra.run.smithi139.stdout:Suggested packages: 2024-09-16T04:26:47.445 INFO:teuthology.orchestra.run.smithi139.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T04:26:47.445 INFO:teuthology.orchestra.run.smithi139.stdout:Recommended packages: 2024-09-16T04:26:47.445 INFO:teuthology.orchestra.run.smithi139.stdout: thermald 2024-09-16T04:26:47.457 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2024-09-16T04:26:47.457 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.457 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T04:26:47.458 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.459 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages will be upgraded: 2024-09-16T04:26:47.459 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T04:26:47.498 INFO:teuthology.orchestra.run.smithi139.stdout:The following NEW packages will be installed: 2024-09-16T04:26:47.498 INFO:teuthology.orchestra.run.smithi139.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:26:47.498 INFO:teuthology.orchestra.run.smithi139.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T04:26:47.498 INFO:teuthology.orchestra.run.smithi139.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T04:26:47.499 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages will be upgraded: 2024-09-16T04:26:47.500 INFO:teuthology.orchestra.run.smithi139.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T04:26:47.513 INFO:teuthology.orchestra.run.smithi165.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:26:47.513 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 113 MB of archives. 2024-09-16T04:26:47.513 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T04:26:47.513 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T04:26:47.552 INFO:teuthology.orchestra.run.smithi139.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:26:47.553 INFO:teuthology.orchestra.run.smithi139.stdout:Need to get 113 MB of archives. 2024-09-16T04:26:47.553 INFO:teuthology.orchestra.run.smithi139.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T04:26:47.553 INFO:teuthology.orchestra.run.smithi139.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T04:26:47.570 INFO:teuthology.orchestra.run.smithi059.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:26:47.571 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 113 MB of archives. 2024-09-16T04:26:47.571 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T04:26:47.571 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T04:26:47.940 INFO:teuthology.orchestra.run.smithi139.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T04:26:47.952 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T04:26:48.037 INFO:teuthology.orchestra.run.smithi139.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T04:26:48.048 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T04:26:48.697 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T04:26:48.832 INFO:teuthology.orchestra.run.smithi139.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T04:26:48.832 INFO:teuthology.orchestra.run.smithi139.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T04:26:48.832 INFO:teuthology.orchestra.run.smithi139.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T04:26:48.845 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T04:26:48.845 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T04:26:48.845 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T04:26:48.949 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T04:26:48.965 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T04:26:48.971 INFO:teuthology.orchestra.run.smithi139.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T04:26:48.986 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T04:26:49.004 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T04:26:49.093 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T04:26:49.362 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T04:26:49.375 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 113 MB in 2s (74.5 MB/s) 2024-09-16T04:26:49.398 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 113 MB in 1s (77.3 MB/s) 2024-09-16T04:26:49.568 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T04:26:49.581 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T04:26:51.219 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T04:26:51.219 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T04:26:51.219 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T04:26:51.476 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T04:26:51.481 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:51.518 INFO:teuthology.orchestra.run.smithi139.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T04:26:51.522 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:51.575 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T04:26:51.604 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T04:26:51.605 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T04:26:51.605 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T04:26:51.628 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:51.656 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T04:26:51.676 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:51.970 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T04:26:52.055 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T04:26:52.056 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 113 MB in 5s (25.0 MB/s) 2024-09-16T04:26:52.261 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T04:26:52.454 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 113 MB in 5s (24.3 MB/s) 2024-09-16T04:26:52.625 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T04:26:53.795 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T04:26:53.817 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:53.904 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:54.254 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T04:26:54.276 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:54.340 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T04:26:54.360 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:54.372 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:54.398 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:54.429 INFO:teuthology.orchestra.run.smithi144.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T04:26:54.433 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:54.592 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T04:26:54.596 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:54.655 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:54.783 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:54.824 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T04:26:54.845 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:54.899 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:57.001 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T04:26:57.023 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:57.120 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:57.232 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T04:26:57.251 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:57.339 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:57.771 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T04:26:57.792 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:57.808 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T04:26:57.829 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:26:57.829 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:26:57.866 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:00.900 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:01.027 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:01.385 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:01.500 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:01.506 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:01.599 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:01.784 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T04:27:01.796 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T04:27:01.834 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:01.933 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:02.128 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:02.431 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T04:27:02.453 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T04:27:02.497 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:04.007 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:04.133 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:04.475 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:04.603 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:04.890 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T04:27:04.901 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T04:27:04.931 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:05.131 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:05.242 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:05.600 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:05.728 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:06.073 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T04:27:06.095 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T04:27:06.131 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:07.258 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T04:27:07.296 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:27:07.341 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:08.285 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T04:27:08.322 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:27:08.360 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:10.052 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:10.166 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:10.516 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:10.638 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:10.680 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T04:27:10.717 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:27:10.746 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:10.772 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T04:27:10.899 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:11.021 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:11.153 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:11.519 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:11.649 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:11.784 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T04:27:11.886 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:12.230 INFO:teuthology.orchestra.run.smithi165.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:12.231 INFO:teuthology.orchestra.run.smithi165.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:12.231 INFO:teuthology.orchestra.run.smithi165.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:12.231 INFO:teuthology.orchestra.run.smithi165.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:12.377 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:12.582 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T04:27:12.619 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:27:12.657 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:13.215 INFO:teuthology.orchestra.run.smithi139.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:13.215 INFO:teuthology.orchestra.run.smithi139.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:13.215 INFO:teuthology.orchestra.run.smithi139.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:13.215 INFO:teuthology.orchestra.run.smithi139.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:13.397 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:13.468 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:13.589 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:14.437 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:14.865 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:14.994 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:15.006 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:15.120 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T04:27:15.256 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:15.854 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T04:27:15.980 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T04:27:16.106 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:16.168 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T04:27:16.168 INFO:teuthology.orchestra.run.smithi165.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:16.416 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T04:27:16.542 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T04:27:16.613 INFO:teuthology.orchestra.run.smithi059.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:16.613 INFO:teuthology.orchestra.run.smithi059.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:16.613 INFO:teuthology.orchestra.run.smithi059.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:16.613 INFO:teuthology.orchestra.run.smithi059.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:16.643 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:16.680 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:27:16.694 INFO:teuthology.orchestra.run.smithi139.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T04:27:16.695 INFO:teuthology.orchestra.run.smithi139.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:16.759 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:16.829 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:27:17.195 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:27:17.334 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:17.468 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T04:27:17.612 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:18.159 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:18.959 INFO:teuthology.orchestra.run.smithi144.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:18.960 INFO:teuthology.orchestra.run.smithi144.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:18.960 INFO:teuthology.orchestra.run.smithi144.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:18.960 INFO:teuthology.orchestra.run.smithi144.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:19.133 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:19.561 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T04:27:19.686 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T04:27:19.813 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:20.662 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T04:27:20.662 INFO:teuthology.orchestra.run.smithi059.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:20.958 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:22.326 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T04:27:22.477 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T04:27:22.612 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:27:22.703 INFO:teuthology.orchestra.run.smithi144.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T04:27:22.703 INFO:teuthology.orchestra.run.smithi144.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:30.352 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T04:27:30.353 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub' 2024-09-16T04:27:30.362 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:30.385 INFO:teuthology.orchestra.run.smithi165.stdout:Generating grub configuration file ... 2024-09-16T04:27:30.706 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:30.724 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:30.949 INFO:teuthology.orchestra.run.smithi139.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T04:27:30.949 INFO:teuthology.orchestra.run.smithi139.stdout:Sourcing file `/etc/default/grub' 2024-09-16T04:27:30.967 INFO:teuthology.orchestra.run.smithi139.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:30.983 INFO:teuthology.orchestra.run.smithi139.stdout:Generating grub configuration file ... 2024-09-16T04:27:31.066 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:31.071 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:31.147 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:31.153 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:31.251 INFO:teuthology.orchestra.run.smithi139.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:31.266 INFO:teuthology.orchestra.run.smithi139.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:31.316 INFO:teuthology.orchestra.run.smithi165.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:31.317 INFO:teuthology.orchestra.run.smithi165.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:31.317 INFO:teuthology.orchestra.run.smithi165.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:31.359 INFO:teuthology.orchestra.run.smithi165.stdout:done 2024-09-16T04:27:31.584 INFO:teuthology.orchestra.run.smithi139.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:31.590 INFO:teuthology.orchestra.run.smithi139.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:31.654 INFO:teuthology.orchestra.run.smithi139.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:31.659 INFO:teuthology.orchestra.run.smithi139.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:31.816 INFO:teuthology.orchestra.run.smithi139.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:31.816 INFO:teuthology.orchestra.run.smithi139.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:31.816 INFO:teuthology.orchestra.run.smithi139.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:31.824 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2024-09-16T04:27:31.847 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2024-09-16T04:27:31.847 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2024-09-16T04:27:31.847 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2024-09-16T04:27:31.847 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 21 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:31.848 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2024-09-16T04:27:31.848 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T04:27:31.849 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:27:31.849 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:27:31.849 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-16T04:27:31.849 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:27:31.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2024-09-16T04:27:31.866 INFO:teuthology.orchestra.run.smithi139.stdout:done 2024-09-16T04:27:32.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-16T04:27:32.137 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:27:32.146 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:27:32.163 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:27:32.257 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:27:32.387 DEBUG:teuthology.orchestra.run.smithi139:> dpkg -s linux-image-generic 2024-09-16T04:27:32.409 INFO:teuthology.orchestra.run.smithi139.stdout:Package: linux-image-generic 2024-09-16T04:27:32.409 INFO:teuthology.orchestra.run.smithi139.stdout:Status: install ok installed 2024-09-16T04:27:32.409 INFO:teuthology.orchestra.run.smithi139.stdout:Priority: optional 2024-09-16T04:27:32.409 INFO:teuthology.orchestra.run.smithi139.stdout:Section: kernel 2024-09-16T04:27:32.409 INFO:teuthology.orchestra.run.smithi139.stdout:Installed-Size: 21 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: amd64 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Source: linux-meta 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Recommends: thermald 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:32.410 INFO:teuthology.orchestra.run.smithi139.stdout: available. 2024-09-16T04:27:32.410 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T04:27:32.411 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:27:32.411 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:27:32.411 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T04:27:32.411 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:27:32.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get clean 2024-09-16T04:27:32.617 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get update 2024-09-16T04:27:32.754 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:27:32.876 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:27:32.957 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:27:33.038 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:27:33.590 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-16T04:27:33.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:27:33.671 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-16T04:27:33.885 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-16T04:27:33.886 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-16T04:27:34.062 INFO:teuthology.orchestra.run.smithi165.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T04:27:34.062 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:27:34.062 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:27:34.063 INFO:teuthology.orchestra.run.smithi165.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:27:34.063 INFO:teuthology.orchestra.run.smithi165.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:27:34.063 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:27:34.063 INFO:teuthology.orchestra.run.smithi165.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:27:34.063 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:27:34.103 INFO:teuthology.orchestra.run.smithi165.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:27:34.105 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 21 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:34.117 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:34.118 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2024-09-16T04:27:34.118 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:34.118 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:34.118 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2024-09-16T04:27:34.118 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-16T04:27:34.163 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.bPWZwZkQ5I 2024-09-16T04:27:34.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub/grub.cfg /tmp/tmp.bPWZwZkQ5I 2024-09-16T04:27:34.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.bPWZwZkQ5I 2024-09-16T04:27:34.250 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-16T04:27:34.269 DEBUG:teuthology.orchestra.run.smithi139:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:27:34.332 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-16T04:27:34.335 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.bPWZwZkQ5I is 10KB 2024-09-16T04:27:34.347 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.bPWZwZkQ5I 2024-09-16T04:27:34.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:34.406 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:27:34.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T04:27:34.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:34.474 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T04:27:34.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo update-grub 2024-09-16T04:27:34.542 INFO:teuthology.orchestra.run.smithi139.stdout:Building dependency tree... 2024-09-16T04:27:34.542 INFO:teuthology.orchestra.run.smithi139.stdout:Reading state information... 2024-09-16T04:27:34.729 INFO:teuthology.orchestra.run.smithi139.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T04:27:34.729 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:27:34.729 INFO:teuthology.orchestra.run.smithi139.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:27:34.729 INFO:teuthology.orchestra.run.smithi139.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:27:34.729 INFO:teuthology.orchestra.run.smithi139.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:27:34.730 INFO:teuthology.orchestra.run.smithi139.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:27:34.730 INFO:teuthology.orchestra.run.smithi139.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:27:34.730 INFO:teuthology.orchestra.run.smithi139.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:27:34.771 INFO:teuthology.orchestra.run.smithi139.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:27:34.773 DEBUG:teuthology.orchestra.run.smithi139:> dpkg -s linux-image-generic 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Package: linux-image-generic 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Status: install ok installed 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Priority: optional 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Section: kernel 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Installed-Size: 21 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: amd64 2024-09-16T04:27:34.788 INFO:teuthology.orchestra.run.smithi139.stdout:Source: linux-meta 2024-09-16T04:27:34.789 INFO:teuthology.orchestra.run.smithi139.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:34.789 INFO:teuthology.orchestra.run.smithi139.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:34.789 INFO:teuthology.orchestra.run.smithi139.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:34.789 INFO:teuthology.orchestra.run.smithi139.stdout:Recommends: thermald 2024-09-16T04:27:34.789 INFO:teuthology.orchestra.run.smithi139.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:34.789 INFO:teuthology.orchestra.run.smithi139.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:34.789 INFO:teuthology.orchestra.run.smithi139.stdout: available. 2024-09-16T04:27:34.789 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-09-16T04:27:34.833 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.z5uyAO8RqA 2024-09-16T04:27:34.834 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub/grub.cfg /tmp/tmp.z5uyAO8RqA 2024-09-16T04:27:34.890 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.z5uyAO8RqA 2024-09-16T04:27:34.980 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T04:27:34.980 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub' 2024-09-16T04:27:34.991 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:35.009 INFO:teuthology.orchestra.run.smithi059.stdout:Generating grub configuration file ... 2024-09-16T04:27:35.016 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.z5uyAO8RqA is 10KB 2024-09-16T04:27:35.029 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.z5uyAO8RqA 2024-09-16T04:27:35.034 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:35.090 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:27:35.090 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T04:27:35.090 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:35.157 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T04:27:35.157 DEBUG:teuthology.orchestra.run.smithi139:> sudo update-grub 2024-09-16T04:27:35.259 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:35.280 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:35.564 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:35.570 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:35.639 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:35.645 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:35.722 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub' 2024-09-16T04:27:35.723 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:35.725 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-09-16T04:27:35.801 INFO:teuthology.orchestra.run.smithi059.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:35.801 INFO:teuthology.orchestra.run.smithi059.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:35.801 INFO:teuthology.orchestra.run.smithi059.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:35.844 INFO:teuthology.orchestra.run.smithi059.stdout:done 2024-09-16T04:27:35.949 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:35.955 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:36.226 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:36.231 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:36.274 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:36.280 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:36.363 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 21 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:36.395 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:36.396 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:36.396 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2024-09-16T04:27:36.396 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:36.396 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:36.396 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2024-09-16T04:27:36.396 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T04:27:36.396 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:27:36.396 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:27:36.396 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T04:27:36.397 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:27:36.397 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2024-09-16T04:27:36.397 INFO:teuthology.orchestra.run.smithi165.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:36.398 INFO:teuthology.orchestra.run.smithi165.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:36.398 INFO:teuthology.orchestra.run.smithi165.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:36.419 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-09-16T04:27:36.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-09-16T04:27:36.496 INFO:teuthology.orchestra.run.smithi139.stderr:Sourcing file `/etc/default/grub' 2024-09-16T04:27:36.497 INFO:teuthology.orchestra.run.smithi139.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:36.499 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2024-09-16T04:27:36.544 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-09-16T04:27:36.656 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:27:36.663 INFO:teuthology.orchestra.run.smithi139.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:36.666 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:27:36.669 INFO:teuthology.orchestra.run.smithi139.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:36.683 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:27:36.781 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:27:36.913 INFO:teuthology.orchestra.run.smithi139.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:36.919 INFO:teuthology.orchestra.run.smithi139.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:36.960 INFO:teuthology.orchestra.run.smithi139.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:36.965 INFO:teuthology.orchestra.run.smithi139.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:37.086 INFO:teuthology.orchestra.run.smithi139.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:37.086 INFO:teuthology.orchestra.run.smithi139.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:37.086 INFO:teuthology.orchestra.run.smithi139.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:37.106 INFO:teuthology.orchestra.run.smithi139.stderr:done 2024-09-16T04:27:37.114 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2024-09-16T04:27:37.282 INFO:teuthology.orchestra.run.smithi144.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T04:27:37.282 INFO:teuthology.orchestra.run.smithi144.stdout:Sourcing file `/etc/default/grub' 2024-09-16T04:27:37.309 INFO:teuthology.orchestra.run.smithi144.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:37.326 INFO:teuthology.orchestra.run.smithi144.stdout:Generating grub configuration file ... 2024-09-16T04:27:37.668 INFO:teuthology.orchestra.run.smithi144.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:37.690 INFO:teuthology.orchestra.run.smithi144.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:38.022 INFO:teuthology.orchestra.run.smithi144.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:38.031 INFO:teuthology.orchestra.run.smithi144.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:38.092 INFO:teuthology.orchestra.run.smithi144.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:38.098 INFO:teuthology.orchestra.run.smithi144.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:38.143 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-09-16T04:27:38.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:27:38.224 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-09-16T04:27:38.321 INFO:teuthology.orchestra.run.smithi144.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:38.321 INFO:teuthology.orchestra.run.smithi144.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:38.321 INFO:teuthology.orchestra.run.smithi144.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:38.358 INFO:teuthology.orchestra.run.smithi144.stdout:done 2024-09-16T04:27:38.417 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-09-16T04:27:38.417 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-09-16T04:27:38.592 INFO:teuthology.orchestra.run.smithi059.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T04:27:38.592 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:27:38.592 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:27:38.592 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:27:38.592 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:27:38.593 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:27:38.593 INFO:teuthology.orchestra.run.smithi059.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:27:38.593 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:27:38.632 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:27:38.634 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic 2024-09-16T04:27:38.649 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 21 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2024-09-16T04:27:38.650 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:38.651 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:38.651 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2024-09-16T04:27:38.651 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-09-16T04:27:38.697 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.0fIRIM44s0 2024-09-16T04:27:38.697 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub/grub.cfg /tmp/tmp.0fIRIM44s0 2024-09-16T04:27:38.753 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.0fIRIM44s0 2024-09-16T04:27:38.856 DEBUG:teuthology.orchestra.run.smithi144:> dpkg -s linux-image-generic 2024-09-16T04:27:38.868 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.0fIRIM44s0 is 10KB 2024-09-16T04:27:38.878 INFO:teuthology.orchestra.run.smithi144.stdout:Package: linux-image-generic 2024-09-16T04:27:38.878 INFO:teuthology.orchestra.run.smithi144.stdout:Status: install ok installed 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Priority: optional 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Section: kernel 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Installed-Size: 21 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: amd64 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Source: linux-meta 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Recommends: thermald 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:38.879 INFO:teuthology.orchestra.run.smithi144.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:38.880 INFO:teuthology.orchestra.run.smithi144.stdout: available. 2024-09-16T04:27:38.880 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T04:27:38.880 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:27:38.880 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:27:38.880 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-16T04:27:38.880 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:27:38.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get clean 2024-09-16T04:27:38.885 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.0fIRIM44s0 2024-09-16T04:27:38.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:38.945 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:27:38.945 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T04:27:38.945 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:39.013 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T04:27:39.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo update-grub 2024-09-16T04:27:39.081 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get update 2024-09-16T04:27:39.350 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:27:39.350 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:27:39.405 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:27:39.486 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:27:40.253 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub' 2024-09-16T04:27:40.254 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:40.257 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-09-16T04:27:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:40.666 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:40.672 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:40.711 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-16T04:27:40.713 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:40.718 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:40.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:27:40.793 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-16T04:27:40.850 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:40.850 INFO:teuthology.orchestra.run.smithi059.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:40.850 INFO:teuthology.orchestra.run.smithi059.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:40.870 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-09-16T04:27:40.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-09-16T04:27:41.004 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2024-09-16T04:27:41.005 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2024-09-16T04:27:41.193 INFO:teuthology.orchestra.run.smithi144.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T04:27:41.194 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:27:41.194 INFO:teuthology.orchestra.run.smithi144.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:27:41.194 INFO:teuthology.orchestra.run.smithi144.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:27:41.194 INFO:teuthology.orchestra.run.smithi144.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:27:41.194 INFO:teuthology.orchestra.run.smithi144.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:27:41.194 INFO:teuthology.orchestra.run.smithi144.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:27:41.195 INFO:teuthology.orchestra.run.smithi144.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:27:41.238 INFO:teuthology.orchestra.run.smithi144.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:27:41.238 DEBUG:teuthology.orchestra.run.smithi144:> dpkg -s linux-image-generic 2024-09-16T04:27:41.253 INFO:teuthology.orchestra.run.smithi144.stdout:Package: linux-image-generic 2024-09-16T04:27:41.253 INFO:teuthology.orchestra.run.smithi144.stdout:Status: install ok installed 2024-09-16T04:27:41.253 INFO:teuthology.orchestra.run.smithi144.stdout:Priority: optional 2024-09-16T04:27:41.253 INFO:teuthology.orchestra.run.smithi144.stdout:Section: kernel 2024-09-16T04:27:41.253 INFO:teuthology.orchestra.run.smithi144.stdout:Installed-Size: 21 2024-09-16T04:27:41.253 INFO:teuthology.orchestra.run.smithi144.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:27:41.254 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: amd64 2024-09-16T04:27:41.254 INFO:teuthology.orchestra.run.smithi144.stdout:Source: linux-meta 2024-09-16T04:27:41.254 INFO:teuthology.orchestra.run.smithi144.stdout:Version: 5.15.0.121.121 2024-09-16T04:27:41.254 INFO:teuthology.orchestra.run.smithi144.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:27:41.254 INFO:teuthology.orchestra.run.smithi144.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:27:41.255 INFO:teuthology.orchestra.run.smithi144.stdout:Recommends: thermald 2024-09-16T04:27:41.255 INFO:teuthology.orchestra.run.smithi144.stdout:Description: Generic Linux kernel image 2024-09-16T04:27:41.255 INFO:teuthology.orchestra.run.smithi144.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:27:41.255 INFO:teuthology.orchestra.run.smithi144.stdout: available. 2024-09-16T04:27:41.255 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-09-16T04:27:41.300 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.EY5UWVYbez 2024-09-16T04:27:41.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub/grub.cfg /tmp/tmp.EY5UWVYbez 2024-09-16T04:27:41.357 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.EY5UWVYbez 2024-09-16T04:27:41.481 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.EY5UWVYbez is 10KB 2024-09-16T04:27:41.494 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.EY5UWVYbez 2024-09-16T04:27:41.500 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:41.556 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:27:41.556 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T04:27:41.556 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T04:27:41.640 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T04:27:41.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo update-grub 2024-09-16T04:27:43.165 INFO:teuthology.orchestra.run.smithi144.stderr:Sourcing file `/etc/default/grub' 2024-09-16T04:27:43.166 INFO:teuthology.orchestra.run.smithi144.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:27:43.173 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2024-09-16T04:27:43.374 INFO:teuthology.orchestra.run.smithi144.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:27:43.380 INFO:teuthology.orchestra.run.smithi144.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:27:43.655 INFO:teuthology.orchestra.run.smithi144.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:27:43.660 INFO:teuthology.orchestra.run.smithi144.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:27:43.699 INFO:teuthology.orchestra.run.smithi144.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:27:43.705 INFO:teuthology.orchestra.run.smithi144.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:27:43.828 INFO:teuthology.orchestra.run.smithi144.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:27:43.828 INFO:teuthology.orchestra.run.smithi144.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:27:43.828 INFO:teuthology.orchestra.run.smithi144.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:27:43.838 INFO:teuthology.orchestra.run.smithi144.stderr:done 2024-09-16T04:27:43.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2024-09-16T04:28:06.452 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-16T04:28:06.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T04:28:06.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:07.117 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T04:28:07.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-16T04:28:07.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:10.893 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T04:28:10.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-16T04:28:10.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:13.893 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-16T04:28:13.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-16T04:28:13.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:25.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-16T04:28:25.533 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-16T04:28:29.377 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-09-16T04:28:32.445 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-09-16T04:28:34.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T04:28:34.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:34.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-16T04:28:34.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:37.085 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-16T04:28:37.597 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-16T04:28:38.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-16T04:28:38.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:41.437 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-09-16T04:28:41.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-16T04:28:41.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:44.541 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-09-16T04:28:49.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T04:28:49.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:49.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-16T04:28:49.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:53.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-16T04:28:53.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:28:56.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-16T04:28:56.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:29:07.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-16T04:29:08.029 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-16T04:29:14.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-09-16T04:29:22.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T04:29:22.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:29:22.895 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-16T04:29:23.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-16T04:29:23.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:29:23.332 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-09-16T04:29:23.871 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-16T04:29:23.871 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T04:29:23.871 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-09-16T04:29:23.918 INFO:teuthology.orchestra.run.smithi139.stdout:5.15.0-121-generic 2024-09-16T04:29:23.918 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T04:29:23.918 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:29:23.918 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T04:29:23.963 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T04:29:23.963 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-121-generic"... 2024-09-16T04:29:23.963 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-16T04:29:24.011 INFO:teuthology.orchestra.run.smithi165.stdout:5.15.0-121-generic 2024-09-16T04:29:24.011 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T04:29:24.011 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:29:24.011 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T04:29:24.918 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T04:29:24.919 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:29:25.011 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2024-09-16T04:29:25.012 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-16T04:29:25.012 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:29:25.039 DEBUG:teuthology.parallel:result is None 2024-09-16T04:29:25.566 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-09-16T04:29:25.608 DEBUG:teuthology.parallel:result is None 2024-09-16T04:29:29.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-16T04:29:29.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:29:30.401 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-16T04:29:31.333 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-16T04:29:31.333 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-121-generic"... 2024-09-16T04:29:31.333 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-16T04:29:31.378 INFO:teuthology.orchestra.run.smithi144.stdout:5.15.0-121-generic 2024-09-16T04:29:31.378 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T04:29:31.378 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:29:31.378 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T04:29:32.378 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-16T04:29:32.379 DEBUG:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:29:32.531 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2024-09-16T04:29:32.577 DEBUG:teuthology.parallel:result is None 2024-09-16T04:29:53.445 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T04:30:08.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-16T04:30:08.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:30:08.653 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-09-16T04:30:09.370 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-16T04:30:09.370 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T04:30:09.370 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-09-16T04:30:09.417 INFO:teuthology.orchestra.run.smithi059.stdout:5.15.0-121-generic 2024-09-16T04:30:09.417 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T04:30:09.417 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:30:09.417 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T04:30:10.418 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T04:30:10.419 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:30:10.487 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-09-16T04:30:10.518 DEBUG:teuthology.parallel:result is None 2024-09-16T04:30:10.518 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T04:30:10.525 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T04:30:10.525 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:30:10.527 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:30:10.530 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:30:10.533 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:30:10.539 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T04:30:10.589 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T04:30:10.598 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T04:30:10.598 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:30:10.600 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:30:10.603 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:30:10.605 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:30:10.706 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T04:30:10.714 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T04:30:10.714 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:30:10.716 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:30:10.719 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:30:10.721 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:30:10.738 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.744 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.748 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.749 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.749 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.752 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.758 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:30:10.760 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T04:30:10.768 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T04:30:10.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:30:10.800 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:30:10.812 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:30:10.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:30:10.867 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T04:30:11.286 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T04:30:11.286 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:30:11.289 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:30:11.292 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:30:11.297 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:30:11.302 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:30:11.341 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:30:11.389 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:30:11.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:30:11.452 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:30:11.457 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:30:11.505 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:30:11.506 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:30:11.574 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:30:11.580 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:30:11.626 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:30:11.627 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:30:11.696 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:30:11.702 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:30:11.750 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:30:11.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:30:11.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-09-16T04:30:11.824 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-09-16T04:30:11.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2024-09-16T04:30:11.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-09-16T04:30:11.894 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T04:30:11.902 INFO:teuthology.task.internal:Starting timer... 2024-09-16T04:30:11.902 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T04:30:11.913 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T04:30:11.922 DEBUG:teuthology.task.selinux:Excluding smithi059: OS 'ubuntu' does not support SELinux 2024-09-16T04:30:11.922 DEBUG:teuthology.task.selinux:Excluding smithi139: OS 'ubuntu' does not support SELinux 2024-09-16T04:30:11.922 DEBUG:teuthology.task.selinux:Excluding smithi144: OS 'ubuntu' does not support SELinux 2024-09-16T04:30:11.922 DEBUG:teuthology.task.selinux:Excluding smithi165: OS 'ubuntu' does not support SELinux 2024-09-16T04:30:11.922 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T04:30:11.922 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T04:30:11.922 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T04:30:11.923 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T04:30:11.933 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T04:30:11.936 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T04:30:12.043 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T04:30:12.057 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T04:30:12.058 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi139.front.sepia.ceph.com,smithi144.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T04:37:50.339 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-09-16T04:37:50.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-16T04:37:50.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:37:50.454 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-09-16T04:37:50.537 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-16T04:37:50.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-16T04:37:50.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:37:50.620 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-09-16T04:37:50.690 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-16T04:37:50.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-16T04:37:50.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:37:50.781 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-16T04:37:50.853 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-16T04:37:50.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T04:37:50.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:37:50.946 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-16T04:37:51.025 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T04:37:51.026 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T04:37:51.040 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T04:37:51.040 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:37:51.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:37:51.043 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:37:51.044 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:37:51.047 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:37:51.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:37:51.050 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:37:51.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:37:51.073 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T04:37:51.073 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Command line: ntpd -gq 2024-09-16T04:37:51.073 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: ---------------------------------------------------- 2024-09-16T04:37:51.073 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T04:37:51.073 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T04:37:51.073 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: corporation. Support and training for ntp-4 are 2024-09-16T04:37:51.074 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: available at https://www.nwtime.org/support 2024-09-16T04:37:51.074 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: ---------------------------------------------------- 2024-09-16T04:37:51.074 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: proto: precision = 0.046 usec (-24) 2024-09-16T04:37:51.074 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: basedate set to 2022-02-04 2024-09-16T04:37:51.074 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: gps base set to 2022-02-06 (week 2196) 2024-09-16T04:37:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.075 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.075 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.075 INFO:teuthology.orchestra.run.smithi059.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.077 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T04:37:51.077 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T04:37:51.077 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T04:37:51.077 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Listen normally on 3 ens1f0 172.21.15.59:123 2024-09-16T04:37:51.078 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Listen normally on 4 lo [::1]:123 2024-09-16T04:37:51.078 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:192a%4]:123 2024-09-16T04:37:51.078 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:37:51 ntpd[18417]: Listening on routing socket on fd #22 for interface updates 2024-09-16T04:37:51.080 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T04:37:51.080 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Command line: ntpd -gq 2024-09-16T04:37:51.080 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: ---------------------------------------------------- 2024-09-16T04:37:51.080 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: corporation. Support and training for ntp-4 are 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: available at https://www.nwtime.org/support 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: ---------------------------------------------------- 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: proto: precision = 0.063 usec (-24) 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: basedate set to 2022-02-04 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: gps base set to 2022-02-06 (week 2196) 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T04:37:51.081 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Command line: ntpd -gq 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: ---------------------------------------------------- 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: corporation. Support and training for ntp-4 are 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: available at https://www.nwtime.org/support 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: ---------------------------------------------------- 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: proto: precision = 0.059 usec (-24) 2024-09-16T04:37:51.082 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: basedate set to 2022-02-04 2024-09-16T04:37:51.083 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: gps base set to 2022-02-06 (week 2196) 2024-09-16T04:37:51.083 INFO:teuthology.orchestra.run.smithi144.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.083 INFO:teuthology.orchestra.run.smithi139.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.084 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.084 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.085 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.085 INFO:teuthology.orchestra.run.smithi144.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.085 INFO:teuthology.orchestra.run.smithi139.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.087 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T04:37:51.087 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T04:37:51.087 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T04:37:51.087 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Listen normally on 3 enp3s0f1 172.21.15.144:123 2024-09-16T04:37:51.087 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Listen normally on 4 lo [::1]:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f5f%5]:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:37:51 ntpd[18459]: Listening on routing socket on fd #22 for interface updates 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Listen normally on 3 enp3s0f1 172.21.15.139:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Listen normally on 4 lo [::1]:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f59%5]:123 2024-09-16T04:37:51.088 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:51 ntpd[18462]: Listening on routing socket on fd #22 for interface updates 2024-09-16T04:37:51.091 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T04:37:51.091 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Command line: ntpd -gq 2024-09-16T04:37:51.092 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: ---------------------------------------------------- 2024-09-16T04:37:51.092 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T04:37:51.092 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T04:37:51.092 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: corporation. Support and training for ntp-4 are 2024-09-16T04:37:51.092 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: available at https://www.nwtime.org/support 2024-09-16T04:37:51.092 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: ---------------------------------------------------- 2024-09-16T04:37:51.093 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: proto: precision = 0.067 usec (-24) 2024-09-16T04:37:51.093 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: basedate set to 2022-02-04 2024-09-16T04:37:51.093 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: gps base set to 2022-02-06 (week 2196) 2024-09-16T04:37:51.093 INFO:teuthology.orchestra.run.smithi165.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.093 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T04:37:51.094 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.094 INFO:teuthology.orchestra.run.smithi165.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T04:37:51.095 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T04:37:51.095 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T04:37:51.095 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T04:37:51.095 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Listen normally on 3 enp3s0f1 172.21.15.165:123 2024-09-16T04:37:51.096 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Listen normally on 4 lo [::1]:123 2024-09-16T04:37:51.096 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f63%5]:123 2024-09-16T04:37:51.096 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:51 ntpd[18490]: Listening on routing socket on fd #22 for interface updates 2024-09-16T04:37:52.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:52.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:52.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:52 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:52.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:52.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:52 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:52.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:52.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:52 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:52.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:52 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:52.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:52 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:53.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:53 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:53.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:53 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:53.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:53 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:53.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:53 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:53.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:53 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:53.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:53 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:53.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:53 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:53.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:53 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:54.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:54 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:54 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:54 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:54 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:54 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:54.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:54 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:54.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:54 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:54.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:54 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:55.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:55.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:55.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:55.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:55 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:55 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:55 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:55 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:55 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:55 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:55 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:55.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:55 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:55 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:55.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:55 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:55.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:55 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:55 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:55 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:55.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:55 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:55.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:55 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:55.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:55 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:56.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:56 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:56 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:56 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:56 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:56 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:56.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:56 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:56.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:56.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:56.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:56.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:56 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:56.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:56.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:56 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:56.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:56.094 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:56.094 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:56.094 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:56.094 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:56 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:57 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:57.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:57 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:57 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:57 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:57 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:57 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:57.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:57 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:57.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:57 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:57 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:57 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:57 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:57 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:57.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:57 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:57.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:57 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:57 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:57 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:57.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:57 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:57.092 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:57 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:57.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:57 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:58.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:58 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:58 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:58 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:58 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:58 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:58.073 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:58 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:58.076 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:58.077 INFO:teuthology.orchestra.run.smithi139.stdout:16 Sep 04:37:58 ntpd[18462]: ntpd: time slew -0.002294 s 2024-09-16T04:37:58.077 INFO:teuthology.orchestra.run.smithi139.stdout:ntpd: time slew -0.002294s 2024-09-16T04:37:58.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T04:37:58.077 INFO:teuthology.orchestra.run.smithi139.stderr:16 Sep 04:37:58 ntpd[18462]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T04:37:58.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:58.079 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:58 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:58.093 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:58.094 INFO:teuthology.orchestra.run.smithi165.stdout:16 Sep 04:37:58 ntpd[18490]: ntpd: time slew -0.001239 s 2024-09-16T04:37:58.094 INFO:teuthology.orchestra.run.smithi165.stdout:ntpd: time slew -0.001239s 2024-09-16T04:37:58.094 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T04:37:58.095 INFO:teuthology.orchestra.run.smithi165.stderr:16 Sep 04:37:58 ntpd[18490]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T04:37:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:37:58.132 INFO:teuthology.orchestra.run.smithi139.stdout:============================================================================== 2024-09-16T04:37:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:37:58.139 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================== 2024-09-16T04:37:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:37:59.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:59.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:59.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:59.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:59.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:59.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:59.076 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:59.076 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:59.076 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:59.076 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:59.076 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:59.076 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:37:59 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:37:59.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:59 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:59.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:59 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:59.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:59 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:37:59.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:59 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:37:59.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:59 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:37:59.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:37:59 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:38:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:00.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:38:00.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:38:00.075 INFO:teuthology.orchestra.run.smithi059.stdout:16 Sep 04:38:00 ntpd[18417]: ntpd: time slew +0.000713 s 2024-09-16T04:38:00.075 INFO:teuthology.orchestra.run.smithi059.stdout:ntpd: time slew +0.000713s 2024-09-16T04:38:00.075 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T04:38:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:16 Sep 04:38:00 ntpd[18417]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T04:38:00.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:38:00.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:00 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:38:00.134 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:38:00.134 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2024-09-16T04:38:00.134 INFO:teuthology.orchestra.run.smithi059.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:00.134 INFO:teuthology.orchestra.run.smithi059.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:00.135 INFO:teuthology.orchestra.run.smithi059.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:00.135 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T04:38:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T04:38:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T04:38:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T04:38:01.078 INFO:teuthology.orchestra.run.smithi144.stdout:16 Sep 04:38:01 ntpd[18459]: ntpd: time slew +0.000695 s 2024-09-16T04:38:01.078 INFO:teuthology.orchestra.run.smithi144.stdout:ntpd: time slew +0.000695s 2024-09-16T04:38:01.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T04:38:01.078 INFO:teuthology.orchestra.run.smithi144.stderr:16 Sep 04:38:01 ntpd[18459]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T04:38:01.136 INFO:teuthology.orchestra.run.smithi144.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:38:01.136 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================== 2024-09-16T04:38:01.136 INFO:teuthology.orchestra.run.smithi144.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:01.136 INFO:teuthology.orchestra.run.smithi144.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:01.136 INFO:teuthology.orchestra.run.smithi144.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:01.136 INFO:teuthology.orchestra.run.smithi144.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:38:01.137 INFO:teuthology.run_tasks:Running task install... 2024-09-16T04:38:01.145 DEBUG:teuthology.task.install:project ceph 2024-09-16T04:38:01.146 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T04:38:01.146 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T04:38:01.146 INFO:teuthology.task.install:Using flavor: default 2024-09-16T04:38:01.151 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T04:38:01.151 INFO:teuthology.task.install:extra packages: [] 2024-09-16T04:38:01.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-key list | grep Ceph 2024-09-16T04:38:01.152 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-key list | grep Ceph 2024-09-16T04:38:01.153 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-key list | grep Ceph 2024-09-16T04:38:01.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-key list | grep Ceph 2024-09-16T04:38:01.214 INFO:teuthology.orchestra.run.smithi139.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T04:38:01.217 INFO:teuthology.orchestra.run.smithi165.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T04:38:01.220 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T04:38:01.236 INFO:teuthology.orchestra.run.smithi144.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T04:38:01.242 INFO:teuthology.orchestra.run.smithi139.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T04:38:01.243 INFO:teuthology.orchestra.run.smithi139.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T04:38:01.243 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T04:38:01.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:38:01.248 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T04:38:01.249 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T04:38:01.249 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T04:38:01.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:38:01.253 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T04:38:01.253 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T04:38:01.253 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T04:38:01.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:38:01.264 INFO:teuthology.orchestra.run.smithi144.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T04:38:01.264 INFO:teuthology.orchestra.run.smithi144.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T04:38:01.264 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T04:38:01.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:38:01.398 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T04:38:01.398 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:01.401 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T04:38:01.401 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:01.407 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T04:38:01.407 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:01.411 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T04:38:01.412 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:01.514 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:38:01.514 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T04:38:01.519 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:38:01.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T04:38:01.526 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:38:01.526 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T04:38:01.529 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-09-16T04:38:01.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-16T04:38:01.535 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:38:01.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T04:38:01.539 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get update 2024-09-16T04:38:01.549 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get update 2024-09-16T04:38:01.703 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:38:01.766 INFO:teuthology.orchestra.run.smithi139.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T04:38:01.766 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T04:38:01.768 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T04:38:01.773 INFO:teuthology.orchestra.run.smithi144.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T04:38:01.787 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T04:38:01.788 INFO:teuthology.orchestra.run.smithi139.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T04:38:01.789 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T04:38:01.795 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T04:38:01.810 INFO:teuthology.orchestra.run.smithi139.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T04:38:01.810 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T04:38:01.813 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T04:38:01.818 INFO:teuthology.orchestra.run.smithi144.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T04:38:01.822 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:38:01.832 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T04:38:01.834 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:38:01.835 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:38:01.835 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T04:38:01.839 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:38:01.839 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:38:01.839 INFO:teuthology.orchestra.run.smithi139.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T04:38:01.848 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:38:01.848 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:38:01.849 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T04:38:01.859 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T04:38:01.866 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T04:38:01.869 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T04:38:01.877 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T04:38:01.904 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:38:01.907 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:38:01.911 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:38:01.920 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:38:01.987 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:38:01.988 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:38:01.997 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:38:02.004 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:38:02.132 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 29.0 kB in 0s (69.4 kB/s) 2024-09-16T04:38:02.208 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 29.0 kB in 0s (59.6 kB/s) 2024-09-16T04:38:02.215 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 29.0 kB in 0s (58.5 kB/s) 2024-09-16T04:38:02.229 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 29.0 kB in 0s (58.7 kB/s) 2024-09-16T04:38:03.195 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-09-16T04:38:03.211 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:03.246 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-16T04:38:03.246 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-16T04:38:03.262 DEBUG:teuthology.orchestra.run.smithi139:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:03.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:03.277 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-09-16T04:38:03.301 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-16T04:38:03.317 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-16T04:38:03.319 DEBUG:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:38:03.327 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-16T04:38:03.371 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-16T04:38:03.489 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-09-16T04:38:03.489 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-09-16T04:38:03.505 INFO:teuthology.orchestra.run.smithi139.stdout:Building dependency tree... 2024-09-16T04:38:03.505 INFO:teuthology.orchestra.run.smithi139.stdout:Reading state information... 2024-09-16T04:38:03.536 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-16T04:38:03.536 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-16T04:38:03.556 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2024-09-16T04:38:03.557 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2024-09-16T04:38:03.695 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:38:03.696 INFO:teuthology.orchestra.run.smithi139.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:38:03.696 INFO:teuthology.orchestra.run.smithi139.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:38:03.696 INFO:teuthology.orchestra.run.smithi139.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:38:03.696 INFO:teuthology.orchestra.run.smithi139.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:38:03.696 INFO:teuthology.orchestra.run.smithi139.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:38:03.697 INFO:teuthology.orchestra.run.smithi139.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:38:03.698 INFO:teuthology.orchestra.run.smithi139.stdout:The following additional packages will be installed: 2024-09-16T04:38:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T04:38:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T04:38:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T04:38:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T04:38:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.699 INFO:teuthology.orchestra.run.smithi139.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T04:38:03.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet zip 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout:Suggested packages: 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T04:38:03.701 INFO:teuthology.orchestra.run.smithi139.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T04:38:03.702 INFO:teuthology.orchestra.run.smithi139.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T04:38:03.702 INFO:teuthology.orchestra.run.smithi139.stdout:Recommended packages: 2024-09-16T04:38:03.702 INFO:teuthology.orchestra.run.smithi139.stdout: btrfs-tools 2024-09-16T04:38:03.706 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:38:03.707 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:38:03.707 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:38:03.707 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:38:03.707 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:38:03.707 INFO:teuthology.orchestra.run.smithi059.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:38:03.708 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:38:03.708 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2024-09-16T04:38:03.708 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T04:38:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T04:38:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T04:38:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T04:38:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T04:38:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T04:38:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T04:38:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T04:38:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T04:38:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T04:38:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T04:38:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T04:38:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T04:38:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet zip 2024-09-16T04:38:03.712 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2024-09-16T04:38:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T04:38:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T04:38:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T04:38:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T04:38:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T04:38:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T04:38:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T04:38:03.714 INFO:teuthology.orchestra.run.smithi059.stdout:Recommended packages: 2024-09-16T04:38:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: btrfs-tools 2024-09-16T04:38:03.738 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:38:03.738 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:38:03.738 INFO:teuthology.orchestra.run.smithi165.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:38:03.738 INFO:teuthology.orchestra.run.smithi165.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:38:03.740 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:38:03.740 INFO:teuthology.orchestra.run.smithi144.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:38:03.740 INFO:teuthology.orchestra.run.smithi144.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:38:03.740 INFO:teuthology.orchestra.run.smithi144.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:38:03.740 INFO:teuthology.orchestra.run.smithi144.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:38:03.741 INFO:teuthology.orchestra.run.smithi144.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:38:03.741 INFO:teuthology.orchestra.run.smithi144.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:38:03.741 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T04:38:03.742 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T04:38:03.743 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T04:38:03.744 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet zip 2024-09-16T04:38:03.744 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2024-09-16T04:38:03.745 INFO:teuthology.orchestra.run.smithi165.stdout: btrfs-tools 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout:The following additional packages will be installed: 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T04:38:03.747 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T04:38:03.748 INFO:teuthology.orchestra.run.smithi144.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet zip 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout:Suggested packages: 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout:Recommended packages: 2024-09-16T04:38:03.749 INFO:teuthology.orchestra.run.smithi144.stdout: btrfs-tools 2024-09-16T04:38:03.803 INFO:teuthology.orchestra.run.smithi139.stdout:The following NEW packages will be installed: 2024-09-16T04:38:03.803 INFO:teuthology.orchestra.run.smithi139.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T04:38:03.803 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T04:38:03.803 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T04:38:03.803 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T04:38:03.803 INFO:teuthology.orchestra.run.smithi139.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T04:38:03.803 INFO:teuthology.orchestra.run.smithi139.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T04:38:03.804 INFO:teuthology.orchestra.run.smithi139.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T04:38:03.804 INFO:teuthology.orchestra.run.smithi139.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T04:38:03.804 INFO:teuthology.orchestra.run.smithi139.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.805 INFO:teuthology.orchestra.run.smithi139.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T04:38:03.806 INFO:teuthology.orchestra.run.smithi139.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T04:38:03.821 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2024-09-16T04:38:03.821 INFO:teuthology.orchestra.run.smithi059.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T04:38:03.821 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T04:38:03.821 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T04:38:03.821 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T04:38:03.822 INFO:teuthology.orchestra.run.smithi059.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T04:38:03.822 INFO:teuthology.orchestra.run.smithi059.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T04:38:03.823 INFO:teuthology.orchestra.run.smithi059.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T04:38:03.823 INFO:teuthology.orchestra.run.smithi059.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T04:38:03.823 INFO:teuthology.orchestra.run.smithi059.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T04:38:03.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T04:38:03.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T04:38:03.823 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T04:38:03.824 INFO:teuthology.orchestra.run.smithi059.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T04:38:03.846 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2024-09-16T04:38:03.846 INFO:teuthology.orchestra.run.smithi165.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T04:38:03.846 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T04:38:03.846 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T04:38:03.846 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T04:38:03.847 INFO:teuthology.orchestra.run.smithi165.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T04:38:03.847 INFO:teuthology.orchestra.run.smithi165.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T04:38:03.848 INFO:teuthology.orchestra.run.smithi165.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T04:38:03.848 INFO:teuthology.orchestra.run.smithi165.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T04:38:03.848 INFO:teuthology.orchestra.run.smithi165.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T04:38:03.848 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T04:38:03.848 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T04:38:03.848 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.848 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T04:38:03.849 INFO:teuthology.orchestra.run.smithi165.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T04:38:03.851 INFO:teuthology.orchestra.run.smithi144.stdout:The following NEW packages will be installed: 2024-09-16T04:38:03.851 INFO:teuthology.orchestra.run.smithi144.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T04:38:03.851 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T04:38:03.851 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T04:38:03.851 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T04:38:03.852 INFO:teuthology.orchestra.run.smithi144.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T04:38:03.853 INFO:teuthology.orchestra.run.smithi144.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T04:38:03.884 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T04:38:03.884 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 181 MB of archives. 2024-09-16T04:38:03.884 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T04:38:03.884 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T04:38:03.907 INFO:teuthology.orchestra.run.smithi165.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T04:38:03.907 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 181 MB of archives. 2024-09-16T04:38:03.907 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T04:38:03.907 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T04:38:03.908 INFO:teuthology.orchestra.run.smithi144.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T04:38:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:Need to get 181 MB of archives. 2024-09-16T04:38:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T04:38:03.909 INFO:teuthology.orchestra.run.smithi144.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T04:38:03.948 INFO:teuthology.orchestra.run.smithi139.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T04:38:03.948 INFO:teuthology.orchestra.run.smithi139.stdout:Need to get 181 MB of archives. 2024-09-16T04:38:03.948 INFO:teuthology.orchestra.run.smithi139.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T04:38:03.949 INFO:teuthology.orchestra.run.smithi139.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T04:38:03.968 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T04:38:03.971 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T04:38:03.973 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T04:38:03.990 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T04:38:03.991 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T04:38:03.992 INFO:teuthology.orchestra.run.smithi139.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T04:38:03.993 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T04:38:03.996 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T04:38:03.997 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T04:38:03.998 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T04:38:04.000 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T04:38:04.016 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T04:38:04.017 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T04:38:04.051 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T04:38:04.057 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T04:38:04.069 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T04:38:04.073 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T04:38:04.074 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T04:38:04.075 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T04:38:04.075 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T04:38:04.078 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T04:38:04.080 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T04:38:04.083 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T04:38:04.084 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T04:38:04.086 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T04:38:04.088 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T04:38:04.089 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T04:38:04.092 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T04:38:04.092 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T04:38:04.094 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T04:38:04.095 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T04:38:04.096 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T04:38:04.097 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T04:38:04.098 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T04:38:04.099 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T04:38:04.100 INFO:teuthology.orchestra.run.smithi144.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T04:38:04.101 INFO:teuthology.orchestra.run.smithi144.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T04:38:04.106 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T04:38:04.107 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T04:38:04.108 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T04:38:04.109 INFO:teuthology.orchestra.run.smithi144.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T04:38:04.109 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T04:38:04.110 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T04:38:04.110 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T04:38:04.110 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T04:38:04.111 INFO:teuthology.orchestra.run.smithi144.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T04:38:04.111 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T04:38:04.112 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T04:38:04.112 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T04:38:04.112 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T04:38:04.113 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T04:38:04.113 INFO:teuthology.orchestra.run.smithi144.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T04:38:04.114 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T04:38:04.115 INFO:teuthology.orchestra.run.smithi144.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T04:38:04.116 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T04:38:04.118 INFO:teuthology.orchestra.run.smithi144.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T04:38:04.120 INFO:teuthology.orchestra.run.smithi144.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T04:38:04.125 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T04:38:04.125 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T04:38:04.130 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T04:38:04.132 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T04:38:04.132 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T04:38:04.133 INFO:teuthology.orchestra.run.smithi144.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T04:38:04.133 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T04:38:04.134 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T04:38:04.134 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T04:38:04.135 INFO:teuthology.orchestra.run.smithi144.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T04:38:04.135 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T04:38:04.136 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T04:38:04.136 INFO:teuthology.orchestra.run.smithi144.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T04:38:04.137 INFO:teuthology.orchestra.run.smithi144.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T04:38:04.137 INFO:teuthology.orchestra.run.smithi144.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T04:38:04.137 INFO:teuthology.orchestra.run.smithi144.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T04:38:04.138 INFO:teuthology.orchestra.run.smithi144.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T04:38:04.138 INFO:teuthology.orchestra.run.smithi144.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T04:38:04.142 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T04:38:04.145 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T04:38:04.145 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T04:38:04.146 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T04:38:04.146 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T04:38:04.147 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T04:38:04.148 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T04:38:04.148 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T04:38:04.149 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T04:38:04.149 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T04:38:04.152 INFO:teuthology.orchestra.run.smithi144.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T04:38:04.152 INFO:teuthology.orchestra.run.smithi144.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T04:38:04.158 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T04:38:04.160 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T04:38:04.165 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T04:38:04.167 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T04:38:04.168 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T04:38:04.168 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T04:38:04.169 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T04:38:04.169 INFO:teuthology.orchestra.run.smithi144.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T04:38:04.170 INFO:teuthology.orchestra.run.smithi165.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T04:38:04.170 INFO:teuthology.orchestra.run.smithi165.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T04:38:04.171 INFO:teuthology.orchestra.run.smithi165.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T04:38:04.172 INFO:teuthology.orchestra.run.smithi144.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T04:38:04.172 INFO:teuthology.orchestra.run.smithi144.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T04:38:04.173 INFO:teuthology.orchestra.run.smithi144.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T04:38:04.173 INFO:teuthology.orchestra.run.smithi144.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T04:38:04.174 INFO:teuthology.orchestra.run.smithi144.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T04:38:04.175 INFO:teuthology.orchestra.run.smithi144.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T04:38:04.176 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T04:38:04.176 INFO:teuthology.orchestra.run.smithi139.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T04:38:04.176 INFO:teuthology.orchestra.run.smithi144.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T04:38:04.181 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T04:38:04.182 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T04:38:04.183 INFO:teuthology.orchestra.run.smithi165.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T04:38:04.183 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T04:38:04.184 INFO:teuthology.orchestra.run.smithi165.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T04:38:04.184 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T04:38:04.184 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T04:38:04.186 INFO:teuthology.orchestra.run.smithi059.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T04:38:04.186 INFO:teuthology.orchestra.run.smithi144.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T04:38:04.187 INFO:teuthology.orchestra.run.smithi059.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T04:38:04.187 INFO:teuthology.orchestra.run.smithi144.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T04:38:04.192 INFO:teuthology.orchestra.run.smithi059.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T04:38:04.193 INFO:teuthology.orchestra.run.smithi059.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T04:38:04.199 INFO:teuthology.orchestra.run.smithi165.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T04:38:04.204 INFO:teuthology.orchestra.run.smithi144.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T04:38:04.205 INFO:teuthology.orchestra.run.smithi165.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T04:38:04.205 INFO:teuthology.orchestra.run.smithi165.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T04:38:04.205 INFO:teuthology.orchestra.run.smithi165.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T04:38:04.205 INFO:teuthology.orchestra.run.smithi165.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T04:38:04.207 INFO:teuthology.orchestra.run.smithi165.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T04:38:04.209 INFO:teuthology.orchestra.run.smithi059.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T04:38:04.209 INFO:teuthology.orchestra.run.smithi059.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T04:38:04.210 INFO:teuthology.orchestra.run.smithi144.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T04:38:04.210 INFO:teuthology.orchestra.run.smithi144.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T04:38:04.210 INFO:teuthology.orchestra.run.smithi165.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T04:38:04.210 INFO:teuthology.orchestra.run.smithi165.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T04:38:04.211 INFO:teuthology.orchestra.run.smithi059.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T04:38:04.211 INFO:teuthology.orchestra.run.smithi144.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T04:38:04.211 INFO:teuthology.orchestra.run.smithi144.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T04:38:04.212 INFO:teuthology.orchestra.run.smithi144.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T04:38:04.213 INFO:teuthology.orchestra.run.smithi059.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T04:38:04.214 INFO:teuthology.orchestra.run.smithi144.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T04:38:04.215 INFO:teuthology.orchestra.run.smithi144.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T04:38:04.216 INFO:teuthology.orchestra.run.smithi165.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T04:38:04.217 INFO:teuthology.orchestra.run.smithi165.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T04:38:04.219 INFO:teuthology.orchestra.run.smithi059.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T04:38:04.221 INFO:teuthology.orchestra.run.smithi144.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T04:38:04.222 INFO:teuthology.orchestra.run.smithi144.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T04:38:04.234 INFO:teuthology.orchestra.run.smithi165.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T04:38:04.234 INFO:teuthology.orchestra.run.smithi165.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T04:38:04.235 INFO:teuthology.orchestra.run.smithi165.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T04:38:04.237 INFO:teuthology.orchestra.run.smithi165.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T04:38:04.238 INFO:teuthology.orchestra.run.smithi144.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T04:38:04.239 INFO:teuthology.orchestra.run.smithi144.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T04:38:04.240 INFO:teuthology.orchestra.run.smithi144.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T04:38:04.243 INFO:teuthology.orchestra.run.smithi144.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T04:38:04.275 INFO:teuthology.orchestra.run.smithi059.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T04:38:04.276 INFO:teuthology.orchestra.run.smithi059.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T04:38:04.277 INFO:teuthology.orchestra.run.smithi059.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T04:38:04.278 INFO:teuthology.orchestra.run.smithi059.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T04:38:04.279 INFO:teuthology.orchestra.run.smithi059.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T04:38:04.290 INFO:teuthology.orchestra.run.smithi165.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T04:38:04.291 INFO:teuthology.orchestra.run.smithi165.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T04:38:04.292 INFO:teuthology.orchestra.run.smithi165.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T04:38:04.292 INFO:teuthology.orchestra.run.smithi165.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T04:38:04.293 INFO:teuthology.orchestra.run.smithi165.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T04:38:04.295 INFO:teuthology.orchestra.run.smithi144.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T04:38:04.296 INFO:teuthology.orchestra.run.smithi144.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T04:38:04.297 INFO:teuthology.orchestra.run.smithi059.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T04:38:04.298 INFO:teuthology.orchestra.run.smithi144.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T04:38:04.298 INFO:teuthology.orchestra.run.smithi144.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T04:38:04.299 INFO:teuthology.orchestra.run.smithi144.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T04:38:04.300 INFO:teuthology.orchestra.run.smithi059.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T04:38:04.300 INFO:teuthology.orchestra.run.smithi059.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T04:38:04.305 INFO:teuthology.orchestra.run.smithi139.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T04:38:04.310 INFO:teuthology.orchestra.run.smithi165.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T04:38:04.312 INFO:teuthology.orchestra.run.smithi165.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T04:38:04.312 INFO:teuthology.orchestra.run.smithi165.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T04:38:04.314 INFO:teuthology.orchestra.run.smithi059.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T04:38:04.314 INFO:teuthology.orchestra.run.smithi059.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T04:38:04.314 INFO:teuthology.orchestra.run.smithi059.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T04:38:04.315 INFO:teuthology.orchestra.run.smithi059.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T04:38:04.315 INFO:teuthology.orchestra.run.smithi059.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T04:38:04.316 INFO:teuthology.orchestra.run.smithi059.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T04:38:04.316 INFO:teuthology.orchestra.run.smithi144.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T04:38:04.318 INFO:teuthology.orchestra.run.smithi144.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T04:38:04.319 INFO:teuthology.orchestra.run.smithi139.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T04:38:04.319 INFO:teuthology.orchestra.run.smithi144.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T04:38:04.326 INFO:teuthology.orchestra.run.smithi139.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T04:38:04.327 INFO:teuthology.orchestra.run.smithi165.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T04:38:04.327 INFO:teuthology.orchestra.run.smithi165.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T04:38:04.327 INFO:teuthology.orchestra.run.smithi165.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T04:38:04.328 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T04:38:04.328 INFO:teuthology.orchestra.run.smithi165.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T04:38:04.328 INFO:teuthology.orchestra.run.smithi165.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T04:38:04.329 INFO:teuthology.orchestra.run.smithi165.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T04:38:04.329 INFO:teuthology.orchestra.run.smithi059.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T04:38:04.329 INFO:teuthology.orchestra.run.smithi059.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T04:38:04.331 INFO:teuthology.orchestra.run.smithi059.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T04:38:04.334 INFO:teuthology.orchestra.run.smithi144.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T04:38:04.334 INFO:teuthology.orchestra.run.smithi144.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T04:38:04.334 INFO:teuthology.orchestra.run.smithi144.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T04:38:04.335 INFO:teuthology.orchestra.run.smithi144.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T04:38:04.336 INFO:teuthology.orchestra.run.smithi144.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T04:38:04.336 INFO:teuthology.orchestra.run.smithi144.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T04:38:04.341 INFO:teuthology.orchestra.run.smithi059.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T04:38:04.343 INFO:teuthology.orchestra.run.smithi165.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T04:38:04.343 INFO:teuthology.orchestra.run.smithi165.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T04:38:04.344 INFO:teuthology.orchestra.run.smithi059.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T04:38:04.346 INFO:teuthology.orchestra.run.smithi059.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T04:38:04.346 INFO:teuthology.orchestra.run.smithi059.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T04:38:04.346 INFO:teuthology.orchestra.run.smithi165.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T04:38:04.350 INFO:teuthology.orchestra.run.smithi059.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T04:38:04.350 INFO:teuthology.orchestra.run.smithi144.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T04:38:04.350 INFO:teuthology.orchestra.run.smithi144.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T04:38:04.352 INFO:teuthology.orchestra.run.smithi059.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T04:38:04.352 INFO:teuthology.orchestra.run.smithi059.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T04:38:04.353 INFO:teuthology.orchestra.run.smithi059.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T04:38:04.353 INFO:teuthology.orchestra.run.smithi144.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T04:38:04.356 INFO:teuthology.orchestra.run.smithi059.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T04:38:04.360 INFO:teuthology.orchestra.run.smithi165.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T04:38:04.362 INFO:teuthology.orchestra.run.smithi165.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T04:38:04.362 INFO:teuthology.orchestra.run.smithi059.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T04:38:04.362 INFO:teuthology.orchestra.run.smithi059.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T04:38:04.362 INFO:teuthology.orchestra.run.smithi165.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T04:38:04.363 INFO:teuthology.orchestra.run.smithi059.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T04:38:04.364 INFO:teuthology.orchestra.run.smithi059.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T04:38:04.365 INFO:teuthology.orchestra.run.smithi139.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T04:38:04.365 INFO:teuthology.orchestra.run.smithi165.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T04:38:04.365 INFO:teuthology.orchestra.run.smithi059.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T04:38:04.366 INFO:teuthology.orchestra.run.smithi139.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T04:38:04.367 INFO:teuthology.orchestra.run.smithi144.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T04:38:04.367 INFO:teuthology.orchestra.run.smithi165.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T04:38:04.368 INFO:teuthology.orchestra.run.smithi165.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T04:38:04.368 INFO:teuthology.orchestra.run.smithi144.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T04:38:04.368 INFO:teuthology.orchestra.run.smithi165.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T04:38:04.369 INFO:teuthology.orchestra.run.smithi144.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T04:38:04.372 INFO:teuthology.orchestra.run.smithi144.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T04:38:04.373 INFO:teuthology.orchestra.run.smithi139.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T04:38:04.374 INFO:teuthology.orchestra.run.smithi144.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T04:38:04.375 INFO:teuthology.orchestra.run.smithi144.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T04:38:04.376 INFO:teuthology.orchestra.run.smithi144.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T04:38:04.377 INFO:teuthology.orchestra.run.smithi165.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T04:38:04.378 INFO:teuthology.orchestra.run.smithi165.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T04:38:04.378 INFO:teuthology.orchestra.run.smithi059.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T04:38:04.379 INFO:teuthology.orchestra.run.smithi059.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T04:38:04.379 INFO:teuthology.orchestra.run.smithi059.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T04:38:04.380 INFO:teuthology.orchestra.run.smithi165.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T04:38:04.381 INFO:teuthology.orchestra.run.smithi059.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T04:38:04.381 INFO:teuthology.orchestra.run.smithi059.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T04:38:04.384 INFO:teuthology.orchestra.run.smithi144.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T04:38:04.385 INFO:teuthology.orchestra.run.smithi144.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T04:38:04.387 INFO:teuthology.orchestra.run.smithi059.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T04:38:04.387 INFO:teuthology.orchestra.run.smithi059.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T04:38:04.388 INFO:teuthology.orchestra.run.smithi144.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T04:38:04.394 INFO:teuthology.orchestra.run.smithi165.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T04:38:04.395 INFO:teuthology.orchestra.run.smithi165.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T04:38:04.395 INFO:teuthology.orchestra.run.smithi165.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T04:38:04.396 INFO:teuthology.orchestra.run.smithi165.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T04:38:04.397 INFO:teuthology.orchestra.run.smithi165.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T04:38:04.399 INFO:teuthology.orchestra.run.smithi059.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T04:38:04.401 INFO:teuthology.orchestra.run.smithi144.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T04:38:04.402 INFO:teuthology.orchestra.run.smithi059.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T04:38:04.402 INFO:teuthology.orchestra.run.smithi059.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T04:38:04.403 INFO:teuthology.orchestra.run.smithi059.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T04:38:04.403 INFO:teuthology.orchestra.run.smithi144.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T04:38:04.403 INFO:teuthology.orchestra.run.smithi144.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T04:38:04.404 INFO:teuthology.orchestra.run.smithi144.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T04:38:04.404 INFO:teuthology.orchestra.run.smithi059.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T04:38:04.405 INFO:teuthology.orchestra.run.smithi165.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T04:38:04.405 INFO:teuthology.orchestra.run.smithi165.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T04:38:04.405 INFO:teuthology.orchestra.run.smithi144.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T04:38:04.411 INFO:teuthology.orchestra.run.smithi059.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T04:38:04.411 INFO:teuthology.orchestra.run.smithi144.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T04:38:04.411 INFO:teuthology.orchestra.run.smithi144.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T04:38:04.412 INFO:teuthology.orchestra.run.smithi165.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T04:38:04.413 INFO:teuthology.orchestra.run.smithi165.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T04:38:04.413 INFO:teuthology.orchestra.run.smithi165.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T04:38:04.419 INFO:teuthology.orchestra.run.smithi144.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T04:38:04.420 INFO:teuthology.orchestra.run.smithi139.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T04:38:04.420 INFO:teuthology.orchestra.run.smithi144.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T04:38:04.421 INFO:teuthology.orchestra.run.smithi144.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T04:38:04.433 INFO:teuthology.orchestra.run.smithi139.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T04:38:04.470 INFO:teuthology.orchestra.run.smithi165.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T04:38:04.514 INFO:teuthology.orchestra.run.smithi139.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T04:38:04.663 INFO:teuthology.orchestra.run.smithi139.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T04:38:04.665 INFO:teuthology.orchestra.run.smithi139.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T04:38:04.699 INFO:teuthology.orchestra.run.smithi059.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T04:38:04.702 INFO:teuthology.orchestra.run.smithi059.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T04:38:04.743 INFO:teuthology.orchestra.run.smithi139.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T04:38:04.755 INFO:teuthology.orchestra.run.smithi139.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T04:38:04.755 INFO:teuthology.orchestra.run.smithi139.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T04:38:04.784 INFO:teuthology.orchestra.run.smithi139.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T04:38:04.790 INFO:teuthology.orchestra.run.smithi059.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T04:38:04.793 INFO:teuthology.orchestra.run.smithi139.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T04:38:04.794 INFO:teuthology.orchestra.run.smithi139.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T04:38:04.795 INFO:teuthology.orchestra.run.smithi139.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T04:38:04.813 INFO:teuthology.orchestra.run.smithi144.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T04:38:04.814 INFO:teuthology.orchestra.run.smithi139.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T04:38:04.817 INFO:teuthology.orchestra.run.smithi139.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T04:38:04.822 INFO:teuthology.orchestra.run.smithi139.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T04:38:04.824 INFO:teuthology.orchestra.run.smithi165.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T04:38:04.825 INFO:teuthology.orchestra.run.smithi139.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T04:38:04.869 INFO:teuthology.orchestra.run.smithi165.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T04:38:04.907 INFO:teuthology.orchestra.run.smithi139.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T04:38:04.909 INFO:teuthology.orchestra.run.smithi165.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T04:38:04.910 INFO:teuthology.orchestra.run.smithi165.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T04:38:04.910 INFO:teuthology.orchestra.run.smithi139.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T04:38:04.914 INFO:teuthology.orchestra.run.smithi165.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T04:38:04.914 INFO:teuthology.orchestra.run.smithi139.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T04:38:04.916 INFO:teuthology.orchestra.run.smithi165.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T04:38:04.918 INFO:teuthology.orchestra.run.smithi139.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T04:38:04.919 INFO:teuthology.orchestra.run.smithi139.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T04:38:04.920 INFO:teuthology.orchestra.run.smithi139.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T04:38:04.922 INFO:teuthology.orchestra.run.smithi139.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T04:38:04.922 INFO:teuthology.orchestra.run.smithi139.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T04:38:04.923 INFO:teuthology.orchestra.run.smithi139.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T04:38:04.954 INFO:teuthology.orchestra.run.smithi165.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T04:38:04.990 INFO:teuthology.orchestra.run.smithi139.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T04:38:05.006 INFO:teuthology.orchestra.run.smithi139.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T04:38:05.006 INFO:teuthology.orchestra.run.smithi139.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T04:38:05.007 INFO:teuthology.orchestra.run.smithi139.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T04:38:05.012 INFO:teuthology.orchestra.run.smithi139.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T04:38:05.014 INFO:teuthology.orchestra.run.smithi139.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T04:38:05.074 INFO:teuthology.orchestra.run.smithi139.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T04:38:05.075 INFO:teuthology.orchestra.run.smithi139.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T04:38:05.076 INFO:teuthology.orchestra.run.smithi139.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T04:38:05.077 INFO:teuthology.orchestra.run.smithi139.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T04:38:05.155 INFO:teuthology.orchestra.run.smithi139.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T04:38:05.157 INFO:teuthology.orchestra.run.smithi139.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T04:38:05.159 INFO:teuthology.orchestra.run.smithi139.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T04:38:05.160 INFO:teuthology.orchestra.run.smithi139.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T04:38:05.171 INFO:teuthology.orchestra.run.smithi139.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T04:38:05.237 INFO:teuthology.orchestra.run.smithi139.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T04:38:05.238 INFO:teuthology.orchestra.run.smithi139.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T04:38:05.239 INFO:teuthology.orchestra.run.smithi139.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T04:38:05.242 INFO:teuthology.orchestra.run.smithi139.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T04:38:05.246 INFO:teuthology.orchestra.run.smithi139.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T04:38:05.260 INFO:teuthology.orchestra.run.smithi144.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T04:38:05.318 INFO:teuthology.orchestra.run.smithi139.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T04:38:05.321 INFO:teuthology.orchestra.run.smithi139.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T04:38:05.322 INFO:teuthology.orchestra.run.smithi139.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T04:38:05.327 INFO:teuthology.orchestra.run.smithi139.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T04:38:05.328 INFO:teuthology.orchestra.run.smithi139.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T04:38:05.396 INFO:teuthology.orchestra.run.smithi144.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T04:38:05.400 INFO:teuthology.orchestra.run.smithi139.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T04:38:05.404 INFO:teuthology.orchestra.run.smithi139.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T04:38:05.444 INFO:teuthology.orchestra.run.smithi144.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T04:38:05.446 INFO:teuthology.orchestra.run.smithi144.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T04:38:05.487 INFO:teuthology.orchestra.run.smithi144.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T04:38:05.493 INFO:teuthology.orchestra.run.smithi144.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T04:38:05.541 INFO:teuthology.orchestra.run.smithi144.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T04:38:05.569 INFO:teuthology.orchestra.run.smithi139.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T04:38:05.570 INFO:teuthology.orchestra.run.smithi139.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T04:38:05.572 INFO:teuthology.orchestra.run.smithi165.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T04:38:05.573 INFO:teuthology.orchestra.run.smithi139.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T04:38:05.575 INFO:teuthology.orchestra.run.smithi165.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T04:38:05.581 INFO:teuthology.orchestra.run.smithi139.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T04:38:05.582 INFO:teuthology.orchestra.run.smithi139.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T04:38:05.616 INFO:teuthology.orchestra.run.smithi139.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T04:38:05.619 INFO:teuthology.orchestra.run.smithi165.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T04:38:05.645 INFO:teuthology.orchestra.run.smithi139.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T04:38:05.645 INFO:teuthology.orchestra.run.smithi139.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T04:38:05.676 INFO:teuthology.orchestra.run.smithi139.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T04:38:05.677 INFO:teuthology.orchestra.run.smithi139.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T04:38:05.736 INFO:teuthology.orchestra.run.smithi139.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T04:38:05.737 INFO:teuthology.orchestra.run.smithi139.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T04:38:05.738 INFO:teuthology.orchestra.run.smithi139.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T04:38:05.739 INFO:teuthology.orchestra.run.smithi139.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T04:38:05.739 INFO:teuthology.orchestra.run.smithi139.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T04:38:05.740 INFO:teuthology.orchestra.run.smithi139.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T04:38:05.747 INFO:teuthology.orchestra.run.smithi139.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T04:38:05.819 INFO:teuthology.orchestra.run.smithi139.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T04:38:05.822 INFO:teuthology.orchestra.run.smithi139.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T04:38:05.824 INFO:teuthology.orchestra.run.smithi139.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T04:38:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T04:38:05.901 INFO:teuthology.orchestra.run.smithi139.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T04:38:05.908 INFO:teuthology.orchestra.run.smithi139.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T04:38:05.909 INFO:teuthology.orchestra.run.smithi139.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T04:38:05.911 INFO:teuthology.orchestra.run.smithi139.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T04:38:05.921 INFO:teuthology.orchestra.run.smithi139.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T04:38:05.923 INFO:teuthology.orchestra.run.smithi139.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T04:38:05.930 INFO:teuthology.orchestra.run.smithi139.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T04:38:05.982 INFO:teuthology.orchestra.run.smithi139.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T04:38:06.011 INFO:teuthology.orchestra.run.smithi059.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T04:38:06.016 INFO:teuthology.orchestra.run.smithi059.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T04:38:06.034 INFO:teuthology.orchestra.run.smithi139.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T04:38:06.035 INFO:teuthology.orchestra.run.smithi139.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T04:38:06.037 INFO:teuthology.orchestra.run.smithi059.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T04:38:06.064 INFO:teuthology.orchestra.run.smithi139.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T04:38:06.083 INFO:teuthology.orchestra.run.smithi059.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T04:38:06.104 INFO:teuthology.orchestra.run.smithi139.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T04:38:06.104 INFO:teuthology.orchestra.run.smithi139.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T04:38:06.121 INFO:teuthology.orchestra.run.smithi139.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T04:38:06.121 INFO:teuthology.orchestra.run.smithi139.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T04:38:06.122 INFO:teuthology.orchestra.run.smithi139.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T04:38:06.127 INFO:teuthology.orchestra.run.smithi139.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T04:38:06.129 INFO:teuthology.orchestra.run.smithi139.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T04:38:06.272 INFO:teuthology.orchestra.run.smithi139.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T04:38:06.278 INFO:teuthology.orchestra.run.smithi139.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T04:38:06.280 INFO:teuthology.orchestra.run.smithi139.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T04:38:06.342 INFO:teuthology.orchestra.run.smithi059.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T04:38:06.377 INFO:teuthology.orchestra.run.smithi139.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T04:38:06.579 INFO:teuthology.orchestra.run.smithi144.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T04:38:06.589 INFO:teuthology.orchestra.run.smithi144.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T04:38:06.655 INFO:teuthology.orchestra.run.smithi144.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T04:38:06.671 INFO:teuthology.orchestra.run.smithi139.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T04:38:07.381 INFO:teuthology.orchestra.run.smithi059.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T04:38:07.382 INFO:teuthology.orchestra.run.smithi059.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T04:38:07.399 INFO:teuthology.orchestra.run.smithi059.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T04:38:07.461 INFO:teuthology.orchestra.run.smithi059.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T04:38:07.476 INFO:teuthology.orchestra.run.smithi059.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T04:38:07.506 INFO:teuthology.orchestra.run.smithi059.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T04:38:07.593 INFO:teuthology.orchestra.run.smithi059.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T04:38:07.845 INFO:teuthology.orchestra.run.smithi139.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T04:38:07.845 INFO:teuthology.orchestra.run.smithi139.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T04:38:07.862 INFO:teuthology.orchestra.run.smithi139.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T04:38:07.904 INFO:teuthology.orchestra.run.smithi139.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T04:38:07.933 INFO:teuthology.orchestra.run.smithi059.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T04:38:07.933 INFO:teuthology.orchestra.run.smithi059.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T04:38:07.939 INFO:teuthology.orchestra.run.smithi139.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T04:38:07.962 INFO:teuthology.orchestra.run.smithi139.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T04:38:08.065 INFO:teuthology.orchestra.run.smithi139.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T04:38:08.224 INFO:teuthology.orchestra.run.smithi165.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T04:38:08.442 INFO:teuthology.orchestra.run.smithi139.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T04:38:08.442 INFO:teuthology.orchestra.run.smithi139.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T04:38:08.665 INFO:teuthology.orchestra.run.smithi165.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T04:38:08.670 INFO:teuthology.orchestra.run.smithi165.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T04:38:08.714 INFO:teuthology.orchestra.run.smithi165.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T04:38:08.912 INFO:teuthology.orchestra.run.smithi165.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T04:38:09.525 INFO:teuthology.orchestra.run.smithi165.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T04:38:10.339 INFO:teuthology.orchestra.run.smithi059.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T04:38:10.342 INFO:teuthology.orchestra.run.smithi059.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T04:38:10.343 INFO:teuthology.orchestra.run.smithi059.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T04:38:10.795 INFO:teuthology.orchestra.run.smithi059.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T04:38:10.902 INFO:teuthology.orchestra.run.smithi139.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T04:38:10.905 INFO:teuthology.orchestra.run.smithi139.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T04:38:10.906 INFO:teuthology.orchestra.run.smithi139.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T04:38:11.010 INFO:teuthology.orchestra.run.smithi144.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T04:38:11.175 INFO:teuthology.orchestra.run.smithi139.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T04:38:11.332 INFO:teuthology.orchestra.run.smithi144.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T04:38:11.343 INFO:teuthology.orchestra.run.smithi165.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T04:38:11.344 INFO:teuthology.orchestra.run.smithi165.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T04:38:11.348 INFO:teuthology.orchestra.run.smithi144.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T04:38:11.351 INFO:teuthology.orchestra.run.smithi144.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T04:38:11.361 INFO:teuthology.orchestra.run.smithi165.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T04:38:11.417 INFO:teuthology.orchestra.run.smithi165.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T04:38:11.423 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 181 MB in 7s (26.0 MB/s) 2024-09-16T04:38:11.433 INFO:teuthology.orchestra.run.smithi144.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T04:38:11.438 INFO:teuthology.orchestra.run.smithi165.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T04:38:11.441 INFO:teuthology.orchestra.run.smithi165.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T04:38:11.515 INFO:teuthology.orchestra.run.smithi165.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T04:38:11.591 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T04:38:11.691 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T04:38:11.697 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T04:38:11.743 INFO:teuthology.orchestra.run.smithi144.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T04:38:11.766 INFO:teuthology.orchestra.run.smithi165.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T04:38:11.766 INFO:teuthology.orchestra.run.smithi165.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T04:38:11.782 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:38:11.851 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 181 MB in 7s (24.2 MB/s) 2024-09-16T04:38:12.017 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T04:38:12.111 INFO:teuthology.orchestra.run.smithi139.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T04:38:12.116 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T04:38:12.118 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T04:38:12.145 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T04:38:12.185 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:38:12.200 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:38:12.495 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T04:38:12.522 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T04:38:12.553 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:38:12.705 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T04:38:12.731 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T04:38:12.771 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:38:12.881 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T04:38:12.908 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T04:38:12.948 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:38:12.996 INFO:teuthology.orchestra.run.smithi144.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T04:38:12.997 INFO:teuthology.orchestra.run.smithi144.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T04:38:13.012 INFO:teuthology.orchestra.run.smithi144.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T04:38:13.123 INFO:teuthology.orchestra.run.smithi144.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T04:38:13.133 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T04:38:13.161 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:13.162 INFO:teuthology.orchestra.run.smithi144.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T04:38:13.165 INFO:teuthology.orchestra.run.smithi144.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T04:38:13.209 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:13.282 INFO:teuthology.orchestra.run.smithi144.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T04:38:13.401 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T04:38:13.428 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:13.476 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:13.493 INFO:teuthology.orchestra.run.smithi165.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T04:38:13.496 INFO:teuthology.orchestra.run.smithi165.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T04:38:13.497 INFO:teuthology.orchestra.run.smithi165.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T04:38:13.595 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T04:38:13.621 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:13.661 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:13.681 INFO:teuthology.orchestra.run.smithi144.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T04:38:13.682 INFO:teuthology.orchestra.run.smithi144.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T04:38:13.837 INFO:teuthology.orchestra.run.smithi165.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T04:38:13.855 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T04:38:13.881 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:13.921 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:13.998 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T04:38:14.024 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:14.064 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:14.266 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T04:38:14.293 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:14.332 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:14.372 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 181 MB in 10s (18.1 MB/s) 2024-09-16T04:38:14.426 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T04:38:14.453 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T04:38:14.501 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T04:38:14.509 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:38:14.599 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T04:38:14.605 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T04:38:14.677 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T04:38:14.683 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:38:14.703 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T04:38:14.752 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:38:14.820 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librados2. 2024-09-16T04:38:14.847 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:14.887 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:15.011 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T04:38:15.030 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librados2. 2024-09-16T04:38:15.037 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T04:38:15.056 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:15.078 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:38:15.097 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:15.314 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librbd1. 2024-09-16T04:38:15.341 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:15.381 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:15.406 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T04:38:15.432 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T04:38:15.472 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:38:15.516 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librbd1. 2024-09-16T04:38:15.542 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:15.582 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:15.817 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T04:38:15.844 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:15.850 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T04:38:15.877 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:15.884 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:15.917 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:16.018 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T04:38:16.045 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:16.085 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:16.115 INFO:teuthology.orchestra.run.smithi144.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T04:38:16.117 INFO:teuthology.orchestra.run.smithi144.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T04:38:16.118 INFO:teuthology.orchestra.run.smithi144.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T04:38:16.161 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rados. 2024-09-16T04:38:16.185 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:16.253 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:16.328 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T04:38:16.355 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:16.373 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rados. 2024-09-16T04:38:16.386 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:16.394 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:16.531 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T04:38:16.546 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:16.571 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:16.578 INFO:teuthology.orchestra.run.smithi144.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T04:38:16.581 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:16.739 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T04:38:16.765 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:16.797 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:16.808 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T04:38:16.835 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:16.875 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:16.882 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T04:38:16.908 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:16.973 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:17.114 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 181 MB in 13s (14.2 MB/s) 2024-09-16T04:38:17.142 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T04:38:17.169 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T04:38:17.177 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T04:38:17.204 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:17.209 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:38:17.234 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T04:38:17.244 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:17.250 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:17.292 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:17.462 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T04:38:17.529 INFO:teuthology.orchestra.run.smithi144.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T04:38:17.534 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T04:38:17.536 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librados2. 2024-09-16T04:38:17.563 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:17.580 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T04:38:17.603 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:17.603 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T04:38:17.607 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T04:38:17.630 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:17.636 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:38:17.647 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:38:17.670 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:18.015 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T04:38:18.041 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T04:38:18.047 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librbd1. 2024-09-16T04:38:18.049 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T04:38:18.073 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:18.074 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T04:38:18.075 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T04:38:18.081 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:38:18.102 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T04:38:18.114 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:18.122 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:38:18.133 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T04:38:18.352 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T04:38:18.359 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T04:38:18.368 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:18.385 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T04:38:18.411 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:18.417 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T04:38:18.499 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T04:38:18.500 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T04:38:18.515 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:18.527 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T04:38:18.549 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:18.567 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:38:18.662 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T04:38:18.688 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:18.728 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:18.764 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T04:38:18.791 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T04:38:18.830 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:38:19.060 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rados. 2024-09-16T04:38:19.072 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T04:38:19.086 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:19.098 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T04:38:19.108 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T04:38:19.113 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T04:38:19.124 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T04:38:19.127 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:19.139 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:38:19.139 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:19.166 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:38:19.196 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:19.400 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T04:38:19.426 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T04:38:19.446 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T04:38:19.461 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T04:38:19.466 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:38:19.472 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:19.487 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T04:38:19.513 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:19.519 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:38:19.699 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T04:38:19.719 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T04:38:19.726 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:19.734 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T04:38:19.769 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:38:19.773 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T04:38:19.774 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:19.800 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:19.839 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua5.1. 2024-09-16T04:38:19.840 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:19.866 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T04:38:19.906 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:38:20.089 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua5.1. 2024-09-16T04:38:20.092 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T04:38:20.107 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:20.115 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T04:38:20.136 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T04:38:20.142 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:20.155 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:38:20.162 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T04:38:20.200 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-any. 2024-09-16T04:38:20.211 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:38:20.216 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T04:38:20.258 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T04:38:20.403 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T04:38:20.430 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T04:38:20.467 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua-any. 2024-09-16T04:38:20.470 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:38:20.482 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T04:38:20.494 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package zip. 2024-09-16T04:38:20.510 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T04:38:20.525 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T04:38:20.544 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T04:38:20.630 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T04:38:20.657 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T04:38:20.705 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:38:20.752 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package zip. 2024-09-16T04:38:20.768 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T04:38:20.811 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T04:38:20.814 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T04:38:20.840 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T04:38:20.873 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package unzip. 2024-09-16T04:38:20.881 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T04:38:20.899 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T04:38:20.939 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:38:21.089 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package unzip. 2024-09-16T04:38:21.091 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librados2. 2024-09-16T04:38:21.115 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T04:38:21.118 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:21.142 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T04:38:21.147 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:38:21.166 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:21.168 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:21.208 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:21.343 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package luarocks. 2024-09-16T04:38:21.369 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T04:38:21.409 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:38:21.518 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package luarocks. 2024-09-16T04:38:21.527 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T04:38:21.544 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T04:38:21.554 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T04:38:21.585 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:38:21.594 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:38:21.661 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librbd1. 2024-09-16T04:38:21.687 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:21.728 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:21.784 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librgw2. 2024-09-16T04:38:21.811 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:21.842 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:21.846 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T04:38:21.862 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T04:38:21.896 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:38:21.976 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librgw2. 2024-09-16T04:38:22.002 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:22.042 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:22.199 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T04:38:22.205 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T04:38:22.226 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T04:38:22.232 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:22.266 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:38:22.281 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:22.394 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T04:38:22.410 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:22.453 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:22.586 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T04:38:22.627 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua5.1. 2024-09-16T04:38:22.627 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rados. 2024-09-16T04:38:22.627 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:22.635 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:22.637 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T04:38:22.653 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:22.677 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:38:22.700 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:22.789 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T04:38:22.816 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T04:38:22.856 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:38:22.933 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T04:38:22.957 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T04:38:22.979 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua-any. 2024-09-16T04:38:22.995 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T04:38:22.997 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:38:23.030 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T04:38:23.111 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T04:38:23.138 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:23.142 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T04:38:23.158 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:23.178 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:23.192 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:23.241 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package zip. 2024-09-16T04:38:23.267 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T04:38:23.307 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T04:38:23.308 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T04:38:23.335 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:23.375 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:23.447 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T04:38:23.462 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:23.505 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:23.612 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-common. 2024-09-16T04:38:23.618 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package unzip. 2024-09-16T04:38:23.639 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:23.645 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T04:38:23.678 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:23.685 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:38:23.789 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-common. 2024-09-16T04:38:23.816 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:23.846 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:23.883 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T04:38:23.909 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:23.958 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:24.105 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package luarocks. 2024-09-16T04:38:24.131 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T04:38:24.172 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:38:24.328 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T04:38:24.354 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T04:38:24.403 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:38:24.529 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librgw2. 2024-09-16T04:38:24.555 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:24.587 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:24.772 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T04:38:24.799 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T04:38:24.847 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T04:38:24.992 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-base. 2024-09-16T04:38:25.019 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:25.068 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:25.100 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T04:38:25.106 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T04:38:25.115 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:25.121 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:25.150 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:25.164 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:25.235 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-base. 2024-09-16T04:38:25.262 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:25.310 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:25.478 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T04:38:25.504 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T04:38:25.509 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T04:38:25.535 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T04:38:25.544 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:38:25.575 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:38:25.654 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T04:38:25.681 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T04:38:25.721 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:38:25.830 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T04:38:25.846 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T04:38:25.889 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:38:25.897 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T04:38:25.903 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T04:38:25.924 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T04:38:25.929 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:25.964 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:38:25.969 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:26.032 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T04:38:26.059 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T04:38:26.099 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:38:26.191 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T04:38:26.218 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T04:38:26.233 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T04:38:26.258 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:38:26.260 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T04:38:26.291 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:38:26.322 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-common. 2024-09-16T04:38:26.349 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:26.381 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:26.410 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T04:38:26.437 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T04:38:26.477 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:38:26.587 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua5.1. 2024-09-16T04:38:26.603 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T04:38:26.613 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T04:38:26.629 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T04:38:26.661 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:38:26.661 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:38:26.721 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T04:38:26.737 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T04:38:26.771 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:38:26.981 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua-any. 2024-09-16T04:38:27.008 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T04:38:27.048 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T04:38:27.066 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T04:38:27.093 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T04:38:27.114 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T04:38:27.132 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:38:27.141 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T04:38:27.180 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:38:27.300 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package zip. 2024-09-16T04:38:27.315 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T04:38:27.359 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T04:38:27.435 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T04:38:27.462 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T04:38:27.466 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T04:38:27.493 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T04:38:27.502 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T04:38:27.533 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:38:27.637 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package unzip. 2024-09-16T04:38:27.660 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T04:38:27.696 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:38:27.788 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-portend. 2024-09-16T04:38:27.810 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T04:38:27.814 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T04:38:27.837 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T04:38:27.852 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-base. 2024-09-16T04:38:27.854 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T04:38:27.877 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T04:38:27.879 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:27.928 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:28.074 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package luarocks. 2024-09-16T04:38:28.101 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T04:38:28.130 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-portend. 2024-09-16T04:38:28.141 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:38:28.157 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T04:38:28.196 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T04:38:28.299 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T04:38:28.326 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T04:38:28.366 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T04:38:28.524 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librgw2. 2024-09-16T04:38:28.524 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T04:38:28.539 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T04:38:28.550 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:28.551 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T04:38:28.566 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T04:38:28.590 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:28.591 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T04:38:28.606 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:38:28.669 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T04:38:28.695 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T04:38:28.735 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:38:28.893 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T04:38:28.909 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T04:38:28.920 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T04:38:28.935 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T04:38:28.960 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:38:28.975 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:38:29.113 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T04:38:29.117 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T04:38:29.133 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:29.140 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T04:38:29.167 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:29.171 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T04:38:29.305 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T04:38:29.332 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T04:38:29.372 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T04:38:29.461 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T04:38:29.467 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T04:38:29.487 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T04:38:29.488 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T04:38:29.494 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T04:38:29.514 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T04:38:29.519 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:38:29.533 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T04:38:29.554 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:38:29.683 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T04:38:29.710 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T04:38:29.749 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T04:38:29.789 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T04:38:29.815 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T04:38:29.836 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T04:38:29.855 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:38:29.857 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T04:38:29.863 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T04:38:29.883 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:29.903 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:38:29.923 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:30.052 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T04:38:30.079 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T04:38:30.119 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:38:30.141 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T04:38:30.168 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T04:38:30.205 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-mako. 2024-09-16T04:38:30.208 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:38:30.232 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T04:38:30.272 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:38:30.335 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-common. 2024-09-16T04:38:30.361 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:30.401 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:30.405 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-mako. 2024-09-16T04:38:30.432 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T04:38:30.463 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:38:30.511 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T04:38:30.537 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T04:38:30.575 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T04:38:30.586 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T04:38:30.602 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T04:38:30.642 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:38:30.758 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T04:38:30.784 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T04:38:30.816 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:38:30.846 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-portend. 2024-09-16T04:38:30.874 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T04:38:30.903 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T04:38:30.905 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T04:38:30.930 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T04:38:30.969 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:38:31.110 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T04:38:31.137 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T04:38:31.174 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T04:38:31.177 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:38:31.201 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T04:38:31.241 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T04:38:31.272 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webob. 2024-09-16T04:38:31.299 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T04:38:31.339 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:38:31.480 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-webob. 2024-09-16T04:38:31.506 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T04:38:31.543 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T04:38:31.554 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:38:31.570 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T04:38:31.610 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:38:31.700 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T04:38:31.727 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T04:38:31.770 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:38:31.799 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-base. 2024-09-16T04:38:31.825 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:31.874 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:31.896 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T04:38:31.899 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T04:38:31.923 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T04:38:31.926 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T04:38:31.963 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T04:38:31.969 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:38:32.078 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T04:38:32.105 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T04:38:32.144 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:38:32.235 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T04:38:32.262 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T04:38:32.299 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T04:38:32.326 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T04:38:32.326 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:38:32.366 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T04:38:32.439 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-paste. 2024-09-16T04:38:32.466 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T04:38:32.497 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:38:32.512 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T04:38:32.538 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T04:38:32.571 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-paste. 2024-09-16T04:38:32.578 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:38:32.587 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T04:38:32.621 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:38:32.668 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T04:38:32.695 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T04:38:32.735 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:38:32.834 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T04:38:32.861 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T04:38:32.900 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:38:32.916 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T04:38:32.931 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T04:38:32.966 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:38:33.029 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-mako. 2024-09-16T04:38:33.056 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T04:38:33.096 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:38:33.123 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T04:38:33.149 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T04:38:33.153 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T04:38:33.180 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T04:38:33.189 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:38:33.210 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T04:38:33.211 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:38:33.237 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T04:38:33.268 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:38:33.390 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T04:38:33.417 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T04:38:33.449 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:38:33.492 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T04:38:33.508 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T04:38:33.514 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T04:38:33.541 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T04:38:33.542 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:38:33.571 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T04:38:33.581 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T04:38:33.598 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T04:38:33.638 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T04:38:33.735 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T04:38:33.761 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T04:38:33.801 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:38:33.854 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T04:38:33.880 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T04:38:33.892 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T04:38:33.919 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T04:38:33.920 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:38:33.949 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T04:38:33.959 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:38:33.976 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T04:38:34.015 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:38:34.112 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webob. 2024-09-16T04:38:34.139 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T04:38:34.195 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:38:34.223 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T04:38:34.249 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T04:38:34.289 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:38:34.322 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T04:38:34.349 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T04:38:34.388 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T04:38:34.395 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T04:38:34.422 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T04:38:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T04:38:34.515 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T04:38:34.541 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T04:38:34.584 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:38:34.592 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T04:38:34.619 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T04:38:34.659 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T04:38:34.725 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-idna. 2024-09-16T04:38:34.752 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T04:38:34.790 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-idna. 2024-09-16T04:38:34.791 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:38:34.816 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T04:38:34.856 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:38:34.884 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T04:38:34.911 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T04:38:34.945 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-portend. 2024-09-16T04:38:34.950 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:38:34.972 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T04:38:35.012 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T04:38:35.061 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T04:38:35.088 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T04:38:35.117 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T04:38:35.119 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:38:35.133 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T04:38:35.167 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:38:35.248 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T04:38:35.253 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-paste. 2024-09-16T04:38:35.264 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T04:38:35.280 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T04:38:35.298 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T04:38:35.320 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:38:35.405 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-requests. 2024-09-16T04:38:35.420 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-requests. 2024-09-16T04:38:35.429 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T04:38:35.447 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T04:38:35.472 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:38:35.486 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:38:35.597 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T04:38:35.601 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T04:38:35.624 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T04:38:35.627 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T04:38:35.664 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:38:35.667 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:38:35.759 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T04:38:35.786 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T04:38:35.789 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T04:38:35.817 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T04:38:35.825 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:38:35.856 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:38:35.971 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T04:38:35.975 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T04:38:35.998 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T04:38:36.002 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T04:38:36.029 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T04:38:36.042 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:38:36.109 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T04:38:36.124 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:36.167 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:36.316 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T04:38:36.342 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T04:38:36.344 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T04:38:36.371 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T04:38:36.382 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T04:38:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T04:38:36.522 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T04:38:36.549 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:36.588 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:36.685 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T04:38:36.705 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T04:38:36.711 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T04:38:36.732 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T04:38:36.752 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:38:36.772 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:38:36.841 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T04:38:36.862 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T04:38:36.864 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T04:38:36.889 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:36.899 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:38:36.928 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:37.030 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-mako. 2024-09-16T04:38:37.056 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T04:38:37.097 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:38:37.118 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T04:38:37.145 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T04:38:37.177 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T04:38:37.202 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T04:38:37.203 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T04:38:37.243 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T04:38:37.300 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T04:38:37.327 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:37.367 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:37.521 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-idna. 2024-09-16T04:38:37.548 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T04:38:37.587 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:38:37.653 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T04:38:37.680 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T04:38:37.719 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:38:37.729 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T04:38:37.756 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:37.796 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:37.906 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T04:38:37.933 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T04:38:37.973 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:38:38.022 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T04:38:38.049 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T04:38:38.088 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T04:38:38.183 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T04:38:38.210 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:38.217 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T04:38:38.244 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T04:38:38.249 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:38.278 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-requests. 2024-09-16T04:38:38.283 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:38:38.305 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T04:38:38.336 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:38:38.417 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T04:38:38.444 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:38.483 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:38.570 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T04:38:38.597 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T04:38:38.606 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T04:38:38.628 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:38:38.632 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T04:38:38.672 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:38:38.761 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T04:38:38.787 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:38.827 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:38.862 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T04:38:38.889 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:38.889 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-webob. 2024-09-16T04:38:38.916 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T04:38:38.919 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:38.955 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:38:38.966 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T04:38:38.982 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:39.016 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:39.292 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T04:38:39.319 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T04:38:39.361 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:38:39.396 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T04:38:39.423 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:39.462 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:39.515 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T04:38:39.542 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:39.598 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:39.678 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T04:38:39.705 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T04:38:39.745 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:38:39.757 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T04:38:39.785 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T04:38:39.823 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:38:39.846 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph. 2024-09-16T04:38:39.862 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:39.913 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:40.064 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-paste. 2024-09-16T04:38:40.091 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T04:38:40.101 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T04:38:40.129 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T04:38:40.131 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:38:40.168 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T04:38:40.266 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T04:38:40.293 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:40.332 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:40.443 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T04:38:40.469 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T04:38:40.509 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:38:40.529 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T04:38:40.557 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:40.595 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:40.644 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T04:38:40.651 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph. 2024-09-16T04:38:40.671 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:40.678 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:40.702 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:40.718 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:40.812 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T04:38:40.838 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T04:38:40.878 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:38:40.998 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T04:38:41.012 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T04:38:41.026 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:41.039 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:41.063 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package cephadm. 2024-09-16T04:38:41.065 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:41.079 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:41.090 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:41.121 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:41.173 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T04:38:41.200 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T04:38:41.240 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T04:38:41.415 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T04:38:41.424 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T04:38:41.442 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:41.451 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T04:38:41.481 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:41.491 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T04:38:41.535 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T04:38:41.561 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T04:38:41.601 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:38:41.660 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T04:38:41.687 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:41.726 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:41.793 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T04:38:41.809 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:41.851 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:41.895 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cephadm. 2024-09-16T04:38:41.922 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:41.961 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:41.964 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T04:38:41.991 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T04:38:42.031 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T04:38:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T04:38:42.275 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T04:38:42.306 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T04:38:42.350 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-idna. 2024-09-16T04:38:42.377 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T04:38:42.417 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:38:42.601 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T04:38:42.627 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:42.666 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:42.695 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T04:38:42.710 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T04:38:42.745 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:38:42.755 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph. 2024-09-16T04:38:42.781 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:42.813 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:42.982 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-babel. 2024-09-16T04:38:43.009 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:43.032 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-requests. 2024-09-16T04:38:43.075 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T04:38:43.075 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:43.075 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T04:38:43.081 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:38:43.086 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:43.123 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:43.376 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T04:38:43.384 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T04:38:43.402 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T04:38:43.411 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T04:38:43.442 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:38:43.450 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:38:43.502 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T04:38:43.528 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:43.568 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:43.737 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T04:38:43.761 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:43.762 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T04:38:43.789 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:43.804 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:43.845 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:43.880 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-babel. 2024-09-16T04:38:43.908 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:43.948 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:43.996 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package cephadm. 2024-09-16T04:38:44.022 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:44.062 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:44.190 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T04:38:44.192 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T04:38:44.217 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T04:38:44.219 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:44.256 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T04:38:44.258 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:44.266 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T04:38:44.294 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T04:38:44.331 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T04:38:44.332 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:38:44.354 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T04:38:44.389 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T04:38:44.559 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-routes. 2024-09-16T04:38:44.586 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T04:38:44.603 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T04:38:44.625 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:38:44.630 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T04:38:44.670 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:38:44.686 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T04:38:44.709 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T04:38:44.713 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:44.724 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:44.752 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:44.767 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:44.914 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T04:38:44.930 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T04:38:44.945 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T04:38:44.961 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:44.964 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T04:38:44.995 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:45.064 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T04:38:45.091 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T04:38:45.130 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T04:38:45.284 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T04:38:45.311 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:45.350 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:45.400 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-routes. 2024-09-16T04:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T04:38:45.466 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:38:45.739 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-babel. 2024-09-16T04:38:45.747 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T04:38:45.761 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T04:38:45.766 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:45.775 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:45.777 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:45.805 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:45.805 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:45.811 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:46.133 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T04:38:46.159 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T04:38:46.199 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:38:46.241 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T04:38:46.257 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T04:38:46.300 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:38:46.392 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T04:38:46.419 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:46.459 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:46.560 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T04:38:46.587 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:46.627 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:46.786 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T04:38:46.813 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T04:38:46.853 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:38:46.963 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T04:38:46.990 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T04:38:47.016 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T04:38:47.030 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T04:38:47.032 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T04:38:47.065 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:38:47.206 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T04:38:47.233 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T04:38:47.273 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:38:47.349 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-routes. 2024-09-16T04:38:47.377 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T04:38:47.416 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:38:47.479 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph. 2024-09-16T04:38:47.486 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T04:38:47.494 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:47.502 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T04:38:47.536 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:38:47.537 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:47.550 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T04:38:47.577 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T04:38:47.617 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:38:47.769 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T04:38:47.796 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:47.807 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T04:38:47.833 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:47.835 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:47.873 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:47.898 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T04:38:47.926 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T04:38:47.964 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:38:48.277 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T04:38:48.303 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T04:38:48.334 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T04:38:48.341 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:38:48.361 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:48.443 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T04:38:48.468 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:48.470 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:48.510 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:48.929 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package cephadm. 2024-09-16T04:38:48.956 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:48.995 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:49.033 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T04:38:49.062 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:49.073 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T04:38:49.100 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T04:38:49.140 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:38:49.186 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:49.324 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T04:38:49.351 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T04:38:49.390 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T04:38:49.413 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T04:38:49.440 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T04:38:49.480 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T04:38:49.685 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T04:38:49.710 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T04:38:49.712 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T04:38:49.737 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:49.751 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:38:49.776 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:49.784 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T04:38:49.810 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T04:38:49.849 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:38:50.088 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T04:38:50.115 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T04:38:50.122 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T04:38:50.151 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T04:38:50.154 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:38:50.169 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T04:38:50.189 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T04:38:50.197 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T04:38:50.237 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:38:50.457 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T04:38:50.484 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T04:38:50.509 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T04:38:50.537 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T04:38:50.548 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:38:50.555 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T04:38:50.575 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:38:50.582 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T04:38:50.621 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T04:38:50.781 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-babel. 2024-09-16T04:38:50.808 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T04:38:50.839 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:38:50.887 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T04:38:50.915 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T04:38:50.942 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T04:38:50.953 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:38:50.969 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T04:38:51.008 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T04:38:51.117 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T04:38:51.144 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T04:38:51.175 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:38:51.206 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T04:38:51.231 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T04:38:51.234 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:51.259 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T04:38:51.289 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T04:38:51.303 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T04:38:51.319 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T04:38:51.353 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:38:51.358 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:51.445 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T04:38:51.468 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:51.512 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:51.601 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T04:38:51.630 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T04:38:51.648 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T04:38:51.675 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T04:38:51.676 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T04:38:51.714 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T04:38:51.815 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T04:38:51.842 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T04:38:51.873 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T04:38:51.996 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T04:38:52.024 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T04:38:52.034 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T04:38:52.061 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T04:38:52.063 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:38:52.117 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:38:52.176 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-routes. 2024-09-16T04:38:52.203 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T04:38:52.243 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:38:52.345 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T04:38:52.372 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T04:38:52.374 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T04:38:52.402 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T04:38:52.411 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T04:38:52.441 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T04:38:52.554 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T04:38:52.581 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:52.621 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:52.739 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T04:38:52.766 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T04:38:52.769 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T04:38:52.797 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T04:38:52.797 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T04:38:52.805 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:38:52.814 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:52.835 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:38:52.864 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:53.134 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T04:38:53.161 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T04:38:53.200 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:38:53.333 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T04:38:53.361 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T04:38:53.400 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:38:53.512 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T04:38:53.516 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T04:38:53.533 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:53.539 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T04:38:53.574 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:53.578 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T04:38:53.736 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T04:38:53.763 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:38:53.802 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:38:53.906 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T04:38:53.909 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T04:38:53.925 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T04:38:53.934 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T04:38:53.935 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T04:38:53.963 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T04:38:53.967 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:38:53.973 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T04:38:54.002 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:38:54.046 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package jq. 2024-09-16T04:38:54.062 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:38:54.105 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T04:38:54.276 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T04:38:54.292 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T04:38:54.307 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package socat. 2024-09-16T04:38:54.322 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T04:38:54.323 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T04:38:54.326 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:38:54.350 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:38:54.366 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:38:54.380 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:38:54.471 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T04:38:54.498 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T04:38:54.537 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:38:54.624 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package jq. 2024-09-16T04:38:54.629 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T04:38:54.640 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:38:54.656 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T04:38:54.682 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T04:38:54.685 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T04:38:54.695 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T04:38:54.701 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T04:38:54.743 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:38:54.858 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T04:38:54.873 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T04:38:54.908 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:38:54.935 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package socat. 2024-09-16T04:38:54.951 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T04:38:54.993 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:38:55.015 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T04:38:55.042 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T04:38:55.081 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:38:55.089 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-test. 2024-09-16T04:38:55.117 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:55.189 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:55.296 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T04:38:55.313 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T04:38:55.355 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:38:55.419 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T04:38:55.446 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T04:38:55.485 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:38:55.709 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-test. 2024-09-16T04:38:55.737 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:55.775 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:55.904 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T04:38:55.932 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:55.978 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:56.227 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T04:38:56.255 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:56.314 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T04:38:56.342 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T04:38:56.379 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:56.380 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:38:56.717 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T04:38:56.745 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:38:56.783 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:38:57.044 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package jq. 2024-09-16T04:38:57.072 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:38:57.119 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T04:38:57.300 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T04:38:57.315 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T04:38:57.350 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T04:38:57.436 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T04:38:57.448 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package socat. 2024-09-16T04:38:57.463 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:57.474 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T04:38:57.501 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:57.521 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:38:57.602 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T04:38:57.630 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T04:38:57.669 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:38:57.796 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T04:38:57.813 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:57.846 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:57.874 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T04:38:57.890 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T04:38:57.933 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:38:58.006 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T04:38:58.033 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T04:38:58.038 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T04:38:58.066 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:38:58.072 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:38:58.104 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:58.108 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T04:38:58.135 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T04:38:58.166 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:38:58.329 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-test. 2024-09-16T04:38:58.356 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:58.392 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T04:38:58.403 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:58.408 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T04:38:58.419 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T04:38:58.425 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:38:58.466 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:38:58.478 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T04:38:58.483 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T04:38:58.506 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T04:38:58.545 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:38:58.786 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T04:38:58.815 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T04:38:58.853 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:38:58.916 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T04:38:58.943 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T04:38:58.982 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:38:59.071 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T04:38:59.098 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T04:38:59.137 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T04:38:59.390 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T04:38:59.419 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T04:38:59.420 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package pkg-config. 2024-09-16T04:38:59.448 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T04:38:59.448 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:38:59.457 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T04:38:59.485 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T04:38:59.486 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:38:59.524 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:38:59.741 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T04:38:59.757 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T04:38:59.791 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:38:59.819 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T04:38:59.819 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T04:38:59.846 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T04:38:59.847 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T04:38:59.885 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T04:38:59.886 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:39:00.169 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T04:39:00.180 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T04:39:00.196 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T04:39:00.197 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T04:39:00.230 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:39:00.235 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:00.457 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package pkg-config. 2024-09-16T04:39:00.486 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T04:39:00.505 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T04:39:00.515 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:39:00.533 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T04:39:00.571 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:00.724 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T04:39:00.751 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:39:00.768 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T04:39:00.790 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:00.797 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T04:39:00.835 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:39:00.894 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T04:39:00.908 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T04:39:00.910 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:39:00.936 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T04:39:00.952 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:00.974 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:01.102 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T04:39:01.130 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:01.169 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:01.249 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T04:39:01.275 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T04:39:01.280 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T04:39:01.286 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T04:39:01.307 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T04:39:01.313 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:01.313 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T04:39:01.346 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:39:01.352 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:01.505 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T04:39:01.532 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T04:39:01.571 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:39:01.616 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T04:39:01.625 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T04:39:01.644 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T04:39:01.652 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:39:01.680 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-py. 2024-09-16T04:39:01.682 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:01.691 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:39:01.707 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T04:39:01.746 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T04:39:01.909 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T04:39:01.936 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T04:39:01.961 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package jq. 2024-09-16T04:39:01.975 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:39:01.988 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T04:39:02.002 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T04:39:02.027 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T04:39:02.030 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T04:39:02.068 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:02.077 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T04:39:02.103 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T04:39:02.141 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:02.279 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package socat. 2024-09-16T04:39:02.287 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T04:39:02.296 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T04:39:02.315 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T04:39:02.338 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:39:02.345 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:39:02.388 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T04:39:02.417 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T04:39:02.455 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:02.604 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T04:39:02.631 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T04:39:02.670 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:02.674 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T04:39:02.691 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T04:39:02.700 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package pkg-config. 2024-09-16T04:39:02.727 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T04:39:02.733 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:39:02.766 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:39:02.799 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-py. 2024-09-16T04:39:02.828 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T04:39:02.874 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T04:39:02.972 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-toml. 2024-09-16T04:39:03.000 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T04:39:03.028 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T04:39:03.044 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T04:39:03.047 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T04:39:03.053 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-test. 2024-09-16T04:39:03.070 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:03.086 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:39:03.112 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:03.203 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T04:39:03.231 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T04:39:03.269 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:03.325 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T04:39:03.353 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T04:39:03.391 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:03.489 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T04:39:03.516 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T04:39:03.555 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:03.712 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T04:39:03.739 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T04:39:03.740 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T04:39:03.768 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T04:39:03.778 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:03.798 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:03.858 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T04:39:03.885 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T04:39:03.926 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:04.055 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T04:39:04.071 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T04:39:04.101 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-toml. 2024-09-16T04:39:04.105 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:04.129 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T04:39:04.167 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T04:39:04.244 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T04:39:04.272 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T04:39:04.311 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:04.513 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T04:39:04.541 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T04:39:04.578 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:04.622 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T04:39:04.649 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T04:39:04.688 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:04.826 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package radosgw. 2024-09-16T04:39:04.854 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:04.892 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:04.949 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T04:39:04.977 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T04:39:04.982 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-py. 2024-09-16T04:39:04.998 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T04:39:05.015 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:05.032 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T04:39:05.268 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T04:39:05.285 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T04:39:05.311 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T04:39:05.326 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:05.327 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T04:39:05.358 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T04:39:05.360 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:05.385 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T04:39:05.424 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:05.679 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T04:39:05.695 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:05.720 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T04:39:05.736 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:05.736 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:05.778 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:05.824 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T04:39:05.851 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T04:39:05.889 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:06.032 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:06.055 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package radosgw. 2024-09-16T04:39:06.083 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:06.090 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T04:39:06.117 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T04:39:06.121 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:06.148 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:39:06.200 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-toml. 2024-09-16T04:39:06.228 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T04:39:06.266 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T04:39:06.300 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:39:06.426 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:39:06.477 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T04:39:06.505 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T04:39:06.543 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:39:06.544 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T04:39:06.570 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T04:39:06.602 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:06.686 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T04:39:06.916 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T04:39:06.933 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:06.939 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T04:39:06.948 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T04:39:06.967 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T04:39:06.974 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:06.975 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T04:39:07.005 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:07.014 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:39:07.060 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T04:39:07.262 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:07.291 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T04:39:07.319 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T04:39:07.358 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:07.437 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package pkg-config. 2024-09-16T04:39:07.464 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T04:39:07.501 INFO:teuthology.orchestra.run.smithi139.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T04:39:07.502 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:39:07.521 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:39:07.569 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:07.630 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:39:07.731 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T04:39:07.747 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T04:39:07.789 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:39:07.899 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T04:39:08.086 INFO:teuthology.orchestra.run.smithi139.stdout:Adding system user cephadm....done 2024-09-16T04:39:08.167 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T04:39:08.187 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package radosgw. 2024-09-16T04:39:08.194 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T04:39:08.215 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:08.225 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:08.242 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:39:08.261 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:08.265 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T04:39:08.486 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T04:39:08.514 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T04:39:08.518 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:39:08.553 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:08.695 INFO:teuthology.orchestra.run.smithi059.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T04:39:08.778 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:39:08.856 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T04:39:08.884 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T04:39:08.904 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:39:08.923 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:08.974 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:09.047 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T04:39:09.063 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:09.097 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:09.155 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T04:39:09.234 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T04:39:09.262 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T04:39:09.300 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:09.377 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:09.414 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:39:09.483 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user cephadm....done 2024-09-16T04:39:09.540 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T04:39:09.587 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-py. 2024-09-16T04:39:09.603 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T04:39:09.627 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:39:09.639 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:39:09.670 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T04:39:09.753 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:39:09.850 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:39:09.915 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:39:09.982 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T04:39:10.009 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T04:39:10.020 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T04:39:10.048 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:10.166 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:39:10.261 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T04:39:10.292 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:39:10.367 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T04:39:10.511 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T04:39:10.520 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T04:39:10.539 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T04:39:10.552 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T04:39:10.577 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:10.770 INFO:teuthology.orchestra.run.smithi165.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T04:39:10.788 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:39:10.820 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:39:10.838 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:10.897 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-toml. 2024-09-16T04:39:10.925 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T04:39:10.946 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T04:39:10.963 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T04:39:11.039 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:39:11.225 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T04:39:11.241 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T04:39:11.264 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:39:11.274 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:11.299 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:11.446 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user cephadm....done 2024-09-16T04:39:11.566 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:39:11.602 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:39:11.620 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T04:39:11.641 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T04:39:11.647 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T04:39:11.686 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:11.817 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:11.878 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:39:11.900 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T04:39:11.964 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T04:39:11.992 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T04:39:12.030 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:12.110 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:39:12.129 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:39:12.185 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:39:12.236 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:39:12.247 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:39:12.345 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:39:12.454 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:39:12.478 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:39:12.640 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T04:39:12.705 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:39:12.760 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package radosgw. 2024-09-16T04:39:12.788 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:12.826 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:12.829 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:12.899 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:39:13.034 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T04:39:13.114 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:39:13.129 INFO:teuthology.orchestra.run.smithi139.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T04:39:13.161 INFO:teuthology.orchestra.run.smithi139.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T04:39:13.225 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:39:13.351 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:13.352 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:39:13.415 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:13.612 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T04:39:13.628 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T04:39:13.671 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:13.683 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:39:13.720 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T04:39:13.809 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:39:13.852 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:13.935 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:39:13.978 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T04:39:13.992 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T04:39:14.005 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T04:39:14.061 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:39:14.237 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:39:14.242 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T04:39:14.272 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:39:14.329 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:39:14.369 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T04:39:14.497 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:14.508 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:39:14.636 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T04:39:14.767 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:14.814 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T04:39:14.931 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T04:39:14.949 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:14.960 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T04:39:15.043 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:39:15.161 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T04:39:15.224 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:39:15.302 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:15.333 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:15.401 INFO:teuthology.orchestra.run.smithi144.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T04:39:15.512 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:15.578 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:39:15.634 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:39:15.704 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:39:15.717 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:15.830 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:39:15.843 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T04:39:15.893 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:39:15.931 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:39:16.019 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:16.120 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:39:16.145 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:39:16.174 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:39:16.321 INFO:teuthology.orchestra.run.smithi144.stdout:Adding system user cephadm....done 2024-09-16T04:39:16.387 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:16.438 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T04:39:16.478 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T04:39:16.565 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T04:39:16.596 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T04:39:16.685 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:39:16.730 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T04:39:16.754 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T04:39:16.806 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T04:39:16.832 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:16.836 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:17.030 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T04:39:17.048 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:39:17.165 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T04:39:17.220 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:17.346 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T04:39:17.400 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T04:39:17.458 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:39:17.541 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:39:17.606 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:39:17.667 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T04:39:17.751 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:39:17.784 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:39:17.793 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T04:39:17.865 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:17.910 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:18.045 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:39:18.094 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:39:18.158 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T04:39:18.196 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T04:39:18.301 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:18.337 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:39:18.371 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T04:39:18.479 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:18.569 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T04:39:18.764 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:39:18.797 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T04:39:18.807 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T04:39:19.036 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:39:19.048 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:39:19.073 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:39:19.075 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T04:39:19.295 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:39:19.327 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T04:39:19.421 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:19.425 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:39:19.539 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:39:19.726 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:39:19.822 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T04:39:19.894 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T04:39:20.086 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:39:20.137 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T04:39:20.199 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T04:39:20.320 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:39:20.345 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:39:20.413 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:20.455 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:20.490 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:39:20.572 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:20.706 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T04:39:20.714 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T04:39:20.773 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:39:20.832 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T04:39:20.887 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:39:20.958 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T04:39:20.974 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:39:21.084 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T04:39:21.099 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:39:21.100 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:39:21.180 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:39:21.226 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:39:21.343 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T04:39:21.506 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:39:21.544 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:21.687 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T04:39:21.717 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T04:39:21.749 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:39:21.780 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T04:39:21.795 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T04:39:21.884 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:21.914 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T04:39:22.046 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:22.163 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:39:22.202 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:39:22.265 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:22.330 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:39:22.392 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T04:39:22.481 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:39:22.545 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:39:22.606 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:22.643 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T04:39:22.732 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T04:39:22.733 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T04:39:22.894 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T04:39:22.992 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:39:22.992 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T04:39:23.109 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:39:23.178 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T04:39:23.236 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:39:23.268 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:39:23.304 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T04:39:23.519 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T04:39:23.545 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:23.750 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:39:23.778 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:39:23.805 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T04:39:23.879 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T04:39:23.940 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T04:39:24.055 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:24.069 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:24.139 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:39:24.182 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:39:24.199 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T04:39:24.203 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T04:39:24.265 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:39:24.325 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T04:39:24.426 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T04:39:24.471 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:39:24.475 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:39:24.532 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:39:24.573 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:39:24.682 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:39:24.734 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T04:39:24.735 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T04:39:24.834 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:39:25.000 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:25.002 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T04:39:25.093 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:39:25.120 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T04:39:25.220 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:39:25.270 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:39:25.276 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T04:39:25.320 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:25.371 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T04:39:25.392 INFO:teuthology.orchestra.run.smithi139.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T04:39:25.529 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T04:39:25.552 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:25.646 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:39:25.695 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:39:25.756 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T04:39:25.781 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:39:25.789 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:25.890 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T04:39:26.013 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:39:26.033 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T04:39:26.174 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:39:26.183 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:39:26.264 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T04:39:26.309 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:39:26.384 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T04:39:26.433 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T04:39:26.524 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T04:39:26.559 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T04:39:26.601 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:39:26.626 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T04:39:26.745 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:26.783 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:39:26.810 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:26.913 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:39:26.919 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:39:27.043 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T04:39:27.045 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T04:39:27.173 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:39:27.229 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:39:27.298 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:39:27.344 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:39:27.364 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T04:39:27.416 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:27.470 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T04:39:27.489 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T04:39:27.521 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:39:27.734 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:39:27.746 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:39:27.798 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T04:39:27.860 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:27.872 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:39:27.924 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T04:39:28.174 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:39:28.183 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T04:39:28.262 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:39:28.459 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:39:28.483 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:39:28.494 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T04:39:28.529 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T04:39:28.639 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T04:39:28.737 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:28.759 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:39:28.812 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:39:28.880 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T04:39:28.890 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:28.894 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:39:29.008 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:39:29.020 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:29.109 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T04:39:29.123 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T04:39:29.189 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:29.249 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T04:39:29.346 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:39:29.359 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:39:29.367 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T04:39:29.448 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:29.474 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:39:29.651 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T04:39:29.676 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:39:29.809 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:39:29.832 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:39:29.894 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T04:39:29.952 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:29.969 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T04:39:30.061 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:30.117 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T04:39:30.154 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:30.187 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:39:30.251 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T04:39:30.280 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T04:39:30.353 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T04:39:30.519 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:30.523 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:30.592 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T04:39:30.638 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T04:39:30.645 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:39:30.649 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:30.776 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T04:39:30.834 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T04:39:30.906 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:39:31.052 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:31.093 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T04:39:31.161 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:31.171 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:39:31.251 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:39:31.262 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:31.395 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T04:39:31.464 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:39:31.466 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:31.466 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:31.636 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:31.662 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T04:39:31.732 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T04:39:31.858 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T04:39:31.930 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T04:39:31.945 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:39:31.955 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:32.056 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T04:39:32.073 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:32.117 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T04:39:32.183 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:32.281 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:39:32.309 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:39:32.332 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T04:39:32.399 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:32.402 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:39:32.458 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T04:39:32.508 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:32.618 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:39:32.743 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T04:39:32.764 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:39:32.954 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.036 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T04:39:33.080 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.132 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:33.206 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T04:39:33.286 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T04:39:33.408 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:39:33.412 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T04:39:33.453 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.491 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.538 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T04:39:33.579 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.610 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T04:39:33.616 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.705 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.753 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:39:33.759 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.840 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T04:39:33.848 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.904 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.957 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:33.975 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T04:39:33.988 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:33.988 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:34.030 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:34.148 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:39:34.343 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T04:39:34.476 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:34.482 INFO:teuthology.orchestra.run.smithi139.stdout:Adding group ceph....done 2024-09-16T04:39:34.485 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:34.585 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T04:39:34.602 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:34.611 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:34.729 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T04:39:34.729 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:34.736 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T04:39:34.839 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:39:34.876 INFO:teuthology.orchestra.run.smithi139.stdout:Adding system user ceph....done 2024-09-16T04:39:34.987 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T04:39:34.996 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:35.045 INFO:teuthology.orchestra.run.smithi139.stdout:Setting system user ceph properties....done 2024-09-16T04:39:35.062 INFO:teuthology.orchestra.run.smithi139.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T04:39:35.106 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:35.113 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T04:39:35.205 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T04:39:35.224 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:35.435 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:35.436 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:35.439 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T04:39:35.537 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T04:39:35.741 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T04:39:35.883 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.010 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.017 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T04:39:36.050 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.109 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.118 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T04:39:36.136 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.176 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.235 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.262 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:39:36.361 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.369 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T04:39:36.487 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.613 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:36.628 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T04:39:36.666 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:36.666 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:36.990 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T04:39:37.144 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:37.163 INFO:teuthology.orchestra.run.smithi059.stdout:Adding group ceph....done 2024-09-16T04:39:37.358 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T04:39:37.516 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:37.532 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user ceph....done 2024-09-16T04:39:37.533 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T04:39:37.625 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:37.635 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T04:39:37.675 INFO:teuthology.orchestra.run.smithi059.stdout:Setting system user ceph properties....done 2024-09-16T04:39:37.694 INFO:teuthology.orchestra.run.smithi059.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T04:39:37.751 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:37.838 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T04:39:37.886 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:37.979 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T04:39:38.012 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.020 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.105 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.211 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T04:39:38.231 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.248 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:38.249 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:38.391 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T04:39:38.555 INFO:teuthology.orchestra.run.smithi165.stdout:Adding group ceph....done 2024-09-16T04:39:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.677 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.755 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.794 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.906 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user ceph....done 2024-09-16T04:39:38.920 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T04:39:38.923 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:38.974 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:38.974 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:39.125 INFO:teuthology.orchestra.run.smithi165.stdout:Setting system user ceph properties....done 2024-09-16T04:39:39.142 INFO:teuthology.orchestra.run.smithi165.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T04:39:39.197 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:39.274 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T04:39:39.298 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:39.390 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:39.391 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:39.416 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:39.516 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:39.592 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T04:39:39.626 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:39.627 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T04:39:39.798 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:39.798 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:39.825 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.080 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.109 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.178 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T04:39:40.207 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.218 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.317 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.356 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.435 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.554 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T04:39:40.561 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:40.641 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:40.641 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:40.764 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:40.765 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:40.931 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.138 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.183 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:41.183 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:41.270 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.446 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.572 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.641 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.643 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T04:39:41.723 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.807 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:41.868 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:41.868 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:41.967 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:42.076 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T04:39:42.093 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:42.177 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T04:39:42.219 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:42.317 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:42.320 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T04:39:42.345 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:42.446 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:42.611 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:42.611 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:42.685 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:42.863 INFO:teuthology.orchestra.run.smithi144.stdout:Adding group ceph....done 2024-09-16T04:39:43.060 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:43.116 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:43.155 INFO:teuthology.orchestra.run.smithi144.stdout:Adding system user ceph....done 2024-09-16T04:39:43.186 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:43.330 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:43.330 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:43.337 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:43.341 INFO:teuthology.orchestra.run.smithi144.stdout:Setting system user ceph properties....done 2024-09-16T04:39:43.353 INFO:teuthology.orchestra.run.smithi144.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T04:39:43.446 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T04:39:43.490 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T04:39:43.557 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:43.557 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:43.788 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T04:39:43.801 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.005 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.103 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:44.103 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:44.103 INFO:teuthology.orchestra.run.smithi139.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T04:39:44.109 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:44.181 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.262 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.307 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.388 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.458 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.561 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.687 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.819 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T04:39:44.838 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:44.852 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:44.852 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:39:44.920 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T04:39:45.033 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:45.033 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:45.056 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T04:39:45.332 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:45.490 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:45.632 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:45.698 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T04:39:45.758 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:45.910 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:46.184 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:46.223 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T04:39:46.279 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T04:39:46.438 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:46.438 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T04:39:46.536 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T04:39:46.669 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T04:39:46.879 INFO:teuthology.orchestra.run.smithi059.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T04:39:46.887 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:46.893 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:47.111 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:47.111 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T04:39:47.369 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T04:39:47.595 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:47.874 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:47.874 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T04:39:48.017 INFO:teuthology.orchestra.run.smithi165.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T04:39:48.022 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:48.396 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:48.522 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:48.665 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:48.880 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:48.881 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T04:39:49.341 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:49.484 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:49.610 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:49.761 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T04:39:50.114 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T04:39:50.231 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T04:39:50.626 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T04:39:51.944 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T04:39:52.591 INFO:teuthology.orchestra.run.smithi144.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T04:39:52.598 DEBUG:teuthology.parallel:result is None 2024-09-16T04:39:52.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:39:52.740 DEBUG:teuthology.orchestra.run.smithi059:> dpkg-query -W -f '${Version}' ceph 2024-09-16T04:39:52.761 INFO:teuthology.orchestra.run.smithi059.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:52.762 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:52.762 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T04:39:52.764 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:39:52.898 DEBUG:teuthology.orchestra.run.smithi139:> dpkg-query -W -f '${Version}' ceph 2024-09-16T04:39:52.917 INFO:teuthology.orchestra.run.smithi139.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:52.917 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:52.917 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T04:39:52.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:39:53.070 DEBUG:teuthology.orchestra.run.smithi144:> dpkg-query -W -f '${Version}' ceph 2024-09-16T04:39:53.089 INFO:teuthology.orchestra.run.smithi144.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:53.091 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:53.091 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T04:39:53.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:39:53.236 DEBUG:teuthology.orchestra.run.smithi165:> dpkg-query -W -f '${Version}' ceph 2024-09-16T04:39:53.256 INFO:teuthology.orchestra.run.smithi165.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:53.257 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:39:53.257 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T04:39:53.259 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T04:39:53.259 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:53.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:39:53.279 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:39:53.279 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:39:53.293 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:39:53.293 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:39:53.307 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:39:53.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:39:53.322 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T04:39:53.323 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:53.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:39:53.337 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:39:53.393 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:39:53.394 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:39:53.407 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:39:53.463 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:39:53.463 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:39:53.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:39:53.531 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:39:53.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:39:53.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:39:53.597 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T04:39:53.598 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:53.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:39:53.612 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:39:53.668 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:39:53.669 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:39:53.683 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:39:53.738 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:39:53.739 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:39:53.752 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:39:53.807 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:39:53.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:39:53.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:39:53.874 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T04:39:53.875 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:53.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:39:53.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:39:53.944 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:39:53.944 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:39:53.957 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:39:54.010 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:39:54.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:39:54.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:39:54.079 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:39:54.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:39:54.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:39:54.145 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T04:39:54.226 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T04:39:54.226 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-09-16T04:39:54.229 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /var/log/ceph 2024-09-16T04:39:54.231 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /var/log/ceph 2024-09-16T04:39:54.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /var/log/ceph 2024-09-16T04:39:54.246 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T04:39:54.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:39:54.290 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:39:54.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:39:54.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:39:54.307 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T04:39:54.307 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:39:54.346 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:39:54.349 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:39:54.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:39:54.375 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T04:39:54.375 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T04:39:54.375 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906025', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfoMci92qZ165G6JxDwvmtdC/wRtQ96bicnhUZm7FeyEx/UdlIHSyYx5u4/pewC+Kmn8OxtrnreeJ01cZbxph4=', 'smithi139.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMLWCwfak+9LK8UE4aubQwW/RuPFQviJl5MRUFwJFPs8PzRBIgRcUUyn1Rs4sbQk8X6bMc7TU8NPkUtQHMqQYVA=', 'smithi144.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDi7RBJQj/3kKjeOdiBKn75cafN5UKG45TvOVRfbuQLJRw4JTYJrtd8Hy8Dpo9kOCj09HV/u8fk37woXZGZ5AjQ=', 'smithi165.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFgy3D4vw9464Msud1VfhXL6DNpoVJfPqQzUM9EWTBnKmO5X3d4XDp2GubM9lloafDMbqL0fJGOfjqzFEUvZlzw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T04:39:54.376 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:39:54.402 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:39:54.404 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:39:54.407 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:39:54.413 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:39:54.450 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:39:54.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:39:54.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:39:54.468 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:54.469 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:39:54.510 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:39:54.510 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-09-16T04:39:54.557 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:39:54.557 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:54.557 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-16T04:39:54.557 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:54.557 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-16 04:34:14.909342110 +0000 2024-09-16T04:39:54.557 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-16 04:34:14.665344279 +0000 2024-09-16T04:39:54.557 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-16 04:34:14.665344279 +0000 2024-09-16T04:39:54.558 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-09-16T04:39:54.558 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:39:54.611 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-16T04:39:54.611 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-16T04:39:54.612 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000448607 s, 1.1 MB/s 2024-09-16T04:39:54.613 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:39:54.662 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-09-16T04:39:54.710 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:39:54.711 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:54.711 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 902 Links: 1 2024-09-16T04:39:54.711 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:54.711 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-16 04:34:15.149339976 +0000 2024-09-16T04:39:54.711 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-16 04:34:15.145340012 +0000 2024-09-16T04:39:54.711 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-16 04:34:15.145340012 +0000 2024-09-16T04:39:54.711 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-09-16T04:39:54.712 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:39:54.763 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-16T04:39:54.763 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-16T04:39:54.763 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000358312 s, 1.4 MB/s 2024-09-16T04:39:54.764 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:39:54.815 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-09-16T04:39:54.863 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:39:54.863 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:54.863 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-09-16T04:39:54.864 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:54.864 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-16 04:34:15.877333505 +0000 2024-09-16T04:39:54.864 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-16 04:34:15.629335710 +0000 2024-09-16T04:39:54.864 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-16 04:34:15.629335710 +0000 2024-09-16T04:39:54.864 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-09-16T04:39:54.864 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:39:54.915 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-16T04:39:54.915 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-16T04:39:54.915 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000390239 s, 1.3 MB/s 2024-09-16T04:39:54.917 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:39:54.966 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 933 Links: 1 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-16 04:34:16.429328598 +0000 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-16 04:34:16.145331122 +0000 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-16 04:34:16.145331122 +0000 2024-09-16T04:39:55.013 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-09-16T04:39:55.014 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:39:55.067 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-16T04:39:55.068 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-16T04:39:55.068 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00038534 s, 1.3 MB/s 2024-09-16T04:39:55.069 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:39:55.118 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:55.118 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:39:55.118 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:39:55.124 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:39:55.124 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2024-09-16T04:39:55.172 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:39:55.172 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:55.172 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-09-16T04:39:55.173 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:55.173 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-16 04:34:15.047242262 +0000 2024-09-16T04:39:55.173 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-16 04:34:14.775244693 +0000 2024-09-16T04:39:55.173 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-16 04:34:14.775244693 +0000 2024-09-16T04:39:55.173 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2024-09-16T04:39:55.173 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:39:55.226 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-16T04:39:55.226 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-16T04:39:55.226 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.00044995 s, 1.1 MB/s 2024-09-16T04:39:55.227 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:39:55.277 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2024-09-16T04:39:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:39:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:55.323 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-09-16T04:39:55.323 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:55.324 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-16 04:34:15.547237797 +0000 2024-09-16T04:39:55.324 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-16 04:34:15.287240119 +0000 2024-09-16T04:39:55.324 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-16 04:34:15.287240119 +0000 2024-09-16T04:39:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2024-09-16T04:39:55.324 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:39:55.377 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-16T04:39:55.377 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-16T04:39:55.377 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000367774 s, 1.4 MB/s 2024-09-16T04:39:55.378 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:39:55.428 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2024-09-16T04:39:55.475 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:39:55.475 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:55.475 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-09-16T04:39:55.475 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:55.475 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-16 04:34:16.063233188 +0000 2024-09-16T04:39:55.476 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-16 04:34:15.799235546 +0000 2024-09-16T04:39:55.476 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-16 04:34:15.799235546 +0000 2024-09-16T04:39:55.476 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2024-09-16T04:39:55.476 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:39:55.530 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-16T04:39:55.530 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-16T04:39:55.530 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000435397 s, 1.2 MB/s 2024-09-16T04:39:55.532 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:39:55.581 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2024-09-16T04:39:55.627 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:39:55.627 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:55.627 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-09-16T04:39:55.627 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:55.627 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-16 04:34:16.499229293 +0000 2024-09-16T04:39:55.627 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-16 04:34:16.271231330 +0000 2024-09-16T04:39:55.628 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-16 04:34:16.271231330 +0000 2024-09-16T04:39:55.628 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2024-09-16T04:39:55.628 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:39:55.681 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-16T04:39:55.681 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-16T04:39:55.681 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000378307 s, 1.4 MB/s 2024-09-16T04:39:55.683 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:39:55.734 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:55.734 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:39:55.734 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:39:55.740 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:39:55.740 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2024-09-16T04:39:55.789 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:39:55.789 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:55.790 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-09-16T04:39:55.790 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:55.790 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-16 04:34:16.369986275 +0000 2024-09-16T04:39:55.790 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-16 04:34:16.089988775 +0000 2024-09-16T04:39:55.790 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-16 04:34:16.089988775 +0000 2024-09-16T04:39:55.790 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-16T04:39:55.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:39:55.843 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-16T04:39:55.843 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-16T04:39:55.843 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000461033 s, 1.1 MB/s 2024-09-16T04:39:55.844 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:39:55.894 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2024-09-16T04:39:55.940 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:39:55.941 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:55.941 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-16T04:39:55.941 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:55.941 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-16 04:34:16.873981773 +0000 2024-09-16T04:39:55.941 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-16 04:34:16.589984310 +0000 2024-09-16T04:39:55.941 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-16 04:34:16.589984310 +0000 2024-09-16T04:39:55.941 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-16T04:39:55.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:39:55.994 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-16T04:39:55.995 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-16T04:39:55.995 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00042928 s, 1.2 MB/s 2024-09-16T04:39:55.996 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:39:56.046 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2024-09-16T04:39:56.092 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:39:56.092 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:56.093 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-09-16T04:39:56.093 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:56.093 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-16 04:34:17.377977271 +0000 2024-09-16T04:39:56.093 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-16 04:34:17.125979521 +0000 2024-09-16T04:39:56.093 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-16 04:34:17.125979521 +0000 2024-09-16T04:39:56.093 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-16T04:39:56.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:39:56.148 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-16T04:39:56.148 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-16T04:39:56.148 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00034787 s, 1.5 MB/s 2024-09-16T04:39:56.149 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:39:56.198 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2024-09-16T04:39:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:39:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:56.244 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-09-16T04:39:56.245 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:56.245 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-16 04:34:17.917972448 +0000 2024-09-16T04:39:56.245 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-16 04:34:17.641974913 +0000 2024-09-16T04:39:56.245 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-16 04:34:17.641974913 +0000 2024-09-16T04:39:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-16T04:39:56.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:39:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-16T04:39:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-16T04:39:56.299 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000414446 s, 1.2 MB/s 2024-09-16T04:39:56.300 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:39:56.349 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:56.350 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:39:56.350 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:39:56.356 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:39:56.356 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-09-16T04:39:56.403 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:39:56.404 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:56.404 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-09-16T04:39:56.404 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:56.404 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 04:34:14.864345109 +0000 2024-09-16T04:39:56.404 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 04:34:14.636347146 +0000 2024-09-16T04:39:56.404 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 04:34:14.636347146 +0000 2024-09-16T04:39:56.404 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-16T04:39:56.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:39:56.457 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T04:39:56.457 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T04:39:56.458 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00044164 s, 1.2 MB/s 2024-09-16T04:39:56.459 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:39:56.508 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 04:34:15.376340535 +0000 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 04:34:15.112342894 +0000 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 04:34:15.112342894 +0000 2024-09-16T04:39:56.559 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-16T04:39:56.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:39:56.612 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T04:39:56.613 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T04:39:56.613 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000345225 s, 1.5 MB/s 2024-09-16T04:39:56.614 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:39:56.664 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-09-16T04:39:56.715 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:39:56.715 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:56.715 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-09-16T04:39:56.715 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:56.716 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 04:34:15.868336141 +0000 2024-09-16T04:39:56.716 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 04:34:15.604338499 +0000 2024-09-16T04:39:56.716 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 04:34:15.604338499 +0000 2024-09-16T04:39:56.716 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-16T04:39:56.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T04:39:56.769 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000415202 s, 1.2 MB/s 2024-09-16T04:39:56.770 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:39:56.820 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-09-16T04:39:56.866 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:39:56.866 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:39:56.866 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-09-16T04:39:56.867 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:39:56.867 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 04:34:16.356331782 +0000 2024-09-16T04:39:56.867 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 04:34:16.088334176 +0000 2024-09-16T04:39:56.867 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 04:34:16.088334176 +0000 2024-09-16T04:39:56.867 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-16T04:39:56.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:39:56.921 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T04:39:56.921 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T04:39:56.921 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000344689 s, 1.5 MB/s 2024-09-16T04:39:56.922 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:39:56.974 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:56.974 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T04:39:56.974 INFO:tasks.ceph:Generating config... 2024-09-16T04:39:56.976 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T04:39:56.976 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T04:39:56.976 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T04:39:56.977 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-16T04:39:56.978 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-16T04:39:56.979 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T04:39:56.979 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T04:39:56.979 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2024-09-16T04:39:56.979 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2024-09-16T04:39:56.979 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T04:39:56.979 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T04:39:56.979 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T04:39:56.979 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T04:39:56.979 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T04:39:56.979 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T04:39:56.979 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T04:39:56.980 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T04:39:57.014 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T04:39:57.042 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T04:39:57.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:39:57.177 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.59'), ('mon.b', '172.21.15.139'), ('mon.c', '172.21.15.144')] 2024-09-16T04:39:57.177 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59,172.21.15.139,172.21.15.144', 'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T04:39:57.178 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:57.178 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:39:57.225 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --add b 172.21.15.139 --add c 172.21.15.144 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:57.312 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:57.312 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid bdbec34e-c8aa-4776-9d32-ea336a99655f 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:fsid bdbec34e-c8aa-4776-9d32-ea336a99655f 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-09-16T04:39:57.314055+0000 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-09-16T04:39:57.314055+0000 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 3 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:1: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.b 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:2: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.c 2024-09-16T04:39:57.313 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T04:39:57.314 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:39:57.319 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bdbec34e-c8aa-4776-9d32-ea336a99655f... 2024-09-16T04:39:57.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:39:57.361 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:39:57.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:39:57.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:39:57.389 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.390 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-09-16T04:39:57.391 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.392 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.139,172.21.15.144" 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: mon election default strategy = 3 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = bdbec34e-c8aa-4776-9d32-ea336a99655f 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.393 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 120 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.394 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression algorithm = snappy 2024-09-16T04:39:57.395 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore compression mode = aggressive 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore zero block detection = True 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 20 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 20 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 10 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: osd blocked scrub grace period = 3600 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug reject backfill probability = 0.1 2024-09-16T04:39:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify cached snaps = True 2024-09-16T04:39:57.397 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify missing on start = True 2024-09-16T04:39:57.397 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:39:57.397 INFO:teuthology.orchestra.run.smithi059.stdout: osd max backfills = 3 2024-09-16T04:39:57.397 INFO:teuthology.orchestra.run.smithi059.stdout: osd max markdown count = 1000 2024-09-16T04:39:57.397 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock override recovery settings = True 2024-09-16T04:39:57.397 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue = debug_random 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue cut off = debug_random 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub min interval = 60 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: osd shutdown pgref assert = True 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-16T04:39:57.398 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-16T04:39:57.399 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:39:57.400 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: mon scrub interval = 300 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:39:57.401 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-09-16T04:39:57.402 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-09-16T04:39:57.402 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: chdir = "" 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: auth supported = cephx 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: filestore xattr use omap = true 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: mon clock drift allowed = 1.000 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.403 INFO:teuthology.orchestra.run.smithi165.stdout: osd crush chooseleaf type = 0 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: auth debug = true 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on old message = true 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on bug = true 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: mon pg warn max object skew = 0 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:39:57.404 INFO:teuthology.orchestra.run.smithi165.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default size = 2 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow primary affinity = true 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow pg remap = true 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on no sortbitwise = false 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on too few osds = false 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:39:57.405 INFO:teuthology.orchestra.run.smithi165.stdout: mon_allow_pool_size_one = true 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: osd default data pool replay window = 5 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: mon allow pool delete = true 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: mon cluster log file level = debug 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: debug asserts on shutdown = true 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: mon health detail to clog = false 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: mon host = "172.21.15.59,172.21.15.139,172.21.15.144" 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: mon election default strategy = 3 2024-09-16T04:39:57.406 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = bdbec34e-c8aa-4776-9d32-ea336a99655f 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout:[osd] 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd journal size = 100 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub load threshold = 5.0 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub max interval = 120 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd recover clone overlap = true 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd recovery max chunk = 1048576 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.407 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug shutdown = true 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug op order = true 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify stray on activate = true 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug trim objects = true 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: osd open classes on start = true 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug pg log writeout = true 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: osd map max advance = 10 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: journal zero on create = true 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.408 INFO:teuthology.orchestra.run.smithi165.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: filestore apply finisher threads = 3 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: bdev debug aio = true 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug misdirected ops = true 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: debug monc = 20 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore block size = 96636764160 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore compression algorithm = snappy 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore compression mode = aggressive 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore fsck on mount = True 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore zero block detection = True 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluefs = 20 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluestore = 20 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-16T04:39:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: debug osd = 20 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: debug rocksdb = 10 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd full ratio = 0.9 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: osd blocked scrub grace period = 3600 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug reject backfill probability = 0.1 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify cached snaps = True 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify missing on start = True 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: osd max backfills = 3 2024-09-16T04:39:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: osd max markdown count = 1000 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock override recovery settings = True 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: osd objectstore = bluestore 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: osd op queue = debug_random 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: osd op queue cut off = debug_random 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub min interval = 60 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: osd shutdown pgref assert = True 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr] 2024-09-16T04:39:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: debug mgr = 20 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: mgr/telemetry/nag = false 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout:[mon] 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: debug paxos = 20 2024-09-16T04:39:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon data avail warn = 5 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon mgr mkfs grace = 240 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd reporter subtree level = osd 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd prime pg temp = true 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:39:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: # 1m isn't quite enough 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: mon scrub interval = 300 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout:[client] 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: rgw cache enabled = true 2024-09-16T04:39:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable ops log = true 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable usage log = true 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.a] 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.b] 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.c] 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2024-09-16T04:39:57.415 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.416 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:39:57.417 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: mon health detail to clog = false 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = "172.21.15.59,172.21.15.139,172.21.15.144" 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: mon election default strategy = 3 2024-09-16T04:39:57.418 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = bdbec34e-c8aa-4776-9d32-ea336a99655f 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 120 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2024-09-16T04:39:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug trim objects = true 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2024-09-16T04:39:57.420 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: debug monc = 20 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore block size = 96636764160 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore compression algorithm = snappy 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore compression mode = aggressive 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore fsck on mount = True 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore zero block detection = True 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluefs = 20 2024-09-16T04:39:57.421 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluestore = 20 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 20 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: debug rocksdb = 10 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd full ratio = 0.9 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd blocked scrub grace period = 3600 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug reject backfill probability = 0.1 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify cached snaps = True 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify missing on start = True 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd max backfills = 3 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd max markdown count = 1000 2024-09-16T04:39:57.422 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock override recovery settings = True 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: osd objectstore = bluestore 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: osd op queue = debug_random 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: osd op queue cut off = debug_random 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub min interval = 60 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: osd shutdown pgref assert = True 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.423 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2024-09-16T04:39:57.424 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: # 1m isn't quite enough 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:39:57.425 INFO:teuthology.orchestra.run.smithi144.stdout: mon scrub interval = 300 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.b] 2024-09-16T04:39:57.426 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.c] 2024-09-16T04:39:57.427 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: chdir = "" 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: auth supported = cephx 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: filestore xattr use omap = true 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: mon clock drift allowed = 1.000 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: osd crush chooseleaf type = 0 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: auth debug = true 2024-09-16T04:39:57.428 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on old message = true 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on bug = true 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: mon pg warn max object skew = 0 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default size = 2 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow primary affinity = true 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow pg remap = true 2024-09-16T04:39:57.429 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on no sortbitwise = false 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on too few osds = false 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon_allow_pool_size_one = true 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: osd default data pool replay window = 5 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon allow pool delete = true 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: mon cluster log file level = debug 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: debug asserts on shutdown = true 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: mon health detail to clog = false 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: mon host = "172.21.15.59,172.21.15.139,172.21.15.144" 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: mon election default strategy = 3 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = bdbec34e-c8aa-4776-9d32-ea336a99655f 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout:[osd] 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: osd journal size = 100 2024-09-16T04:39:57.431 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub load threshold = 5.0 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub max interval = 120 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd recover clone overlap = true 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd recovery max chunk = 1048576 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug shutdown = true 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug op order = true 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify stray on activate = true 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug trim objects = true 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd open classes on start = true 2024-09-16T04:39:57.432 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug pg log writeout = true 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: osd map max advance = 10 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: journal zero on create = true 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: filestore apply finisher threads = 3 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: bdev debug aio = true 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug misdirected ops = true 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: debug monc = 20 2024-09-16T04:39:57.433 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore block size = 96636764160 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore compression algorithm = snappy 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore compression mode = aggressive 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore fsck on mount = True 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore zero block detection = True 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: debug bluefs = 20 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: debug bluestore = 20 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: debug osd = 20 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: debug rocksdb = 10 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd full ratio = 0.9 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: osd blocked scrub grace period = 3600 2024-09-16T04:39:57.434 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug reject backfill probability = 0.1 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify cached snaps = True 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify missing on start = True 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd max backfills = 3 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd max markdown count = 1000 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd mclock override recovery settings = True 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd objectstore = bluestore 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd op queue = debug_random 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd op queue cut off = debug_random 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub min interval = 60 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: osd shutdown pgref assert = True 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.435 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr] 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug mgr = 20 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: mgr/telemetry/nag = false 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout:[mon] 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug paxos = 20 2024-09-16T04:39:57.436 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon data avail warn = 5 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon mgr mkfs grace = 240 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd reporter subtree level = osd 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd prime pg temp = true 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:39:57.437 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: # 1m isn't quite enough 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: mon scrub interval = 300 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout:[client] 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: rgw cache enabled = true 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable ops log = true 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable usage log = true 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:39:57.438 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.a] 2024-09-16T04:39:57.439 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.b] 2024-09-16T04:39:57.439 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.c] 2024-09-16T04:39:57.440 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T04:39:57.440 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T04:39:57.524 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T04:39:57.525 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:57.525 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T04:39:57.530 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:57.530 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T04:39:57.577 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:39:57.577 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:57.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:39:57.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:39:57.642 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:39:57.642 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:57.689 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:39:57.689 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:39:57.689 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:39:57.689 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:39:57.711 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:39:57.711 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:57.760 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:39:57.760 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:39:57.760 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:39:57.760 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:39:57.784 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:39:57.784 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:57.832 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:39:57.833 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:39:57.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:39:57.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:39:57.856 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:39:57.856 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:39:57.907 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T04:39:57.908 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T04:39:57.908 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T04:39:57.954 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T04:39:57.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:39:58.001 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:39:58.011 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T04:39:58.011 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T04:39:58.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T04:39:58.089 INFO:teuthology.orchestra.run.smithi165.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T04:39:58.099 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T04:39:58.100 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T04:39:58.100 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T04:39:58.116 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:58.116 INFO:tasks.ceph:role: osd.0 2024-09-16T04:39:58.116 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:39:58.116 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:39:58.180 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:39:58.180 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:39:58.180 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:39:58.180 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:39:58.180 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:39:58.180 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:39:58.181 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:39:58.181 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:39:58.181 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:39:58.181 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:39:58.653 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-16T04:39:58.656 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-16T04:39:58.656 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T04:39:58.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T04:39:58.802 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:39:58.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:39:58.803 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T04:39:58.860 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:58.860 INFO:tasks.ceph:role: osd.4 2024-09-16T04:39:58.860 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:39:58.860 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:39:58.924 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:39:58.925 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:39:59.348 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-16T04:39:59.350 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-16T04:39:59.350 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T04:39:59.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T04:39:59.465 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:39:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:39:59.466 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-16T04:39:59.519 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T04:39:59.519 INFO:tasks.ceph:role: osd.8 2024-09-16T04:39:59.519 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:39:59.520 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:39:59.583 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:39:59.584 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:39:59.584 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:00.044 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-16T04:40:00.046 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-16T04:40:00.047 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-16T04:40:00.135 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-16T04:40:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:00.144 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:00.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-16T04:40:00.200 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:00.201 INFO:tasks.ceph:role: osd.12 2024-09-16T04:40:00.201 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:40:00.201 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:00.265 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:00.735 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-16T04:40:00.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-16T04:40:00.738 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-16T04:40:00.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-16T04:40:00.893 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:00.894 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:01.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:01.011+0000 7fe88f58e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T04:40:01.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:01.015+0000 7fe88f58e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T04:40:01.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:01.015+0000 7fe88f58e7c0 -1 bdev(0x55592afb1800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:01.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:01.015+0000 7fe88f58e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T04:40:04.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T04:40:04.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:04.255+0000 7fa877c6d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T04:40:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:04.255+0000 7fa877c6d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T04:40:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:04.255+0000 7fa877c6d7c0 -1 bdev(0x55ec1221b800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:04.259+0000 7fa877c6d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T04:40:07.459 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T04:40:07.518 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:07.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:07.603+0000 7fcb2f34d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-16T04:40:07.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:07.603+0000 7fcb2f34d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-16T04:40:07.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:07.603+0000 7fcb2f34d7c0 -1 bdev(0x5583b187f800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:07.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:07.603+0000 7fcb2f34d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-16T04:40:10.789 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-16T04:40:10.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:10.918+0000 7f96515407c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-16T04:40:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:10.918+0000 7f96515407c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-16T04:40:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:10.918+0000 7f96515407c0 -1 bdev(0x563dd8693800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-16T04:40:10.922+0000 7f96515407c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-16T04:40:14.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-16T04:40:14.140 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T04:40:14.155 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:14.155 INFO:tasks.ceph:role: osd.1 2024-09-16T04:40:14.155 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:40:14.156 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:14.219 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:14.220 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:14.680 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-09-16T04:40:14.681 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-09-16T04:40:14.681 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-16T04:40:14.764 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T04:40:14.817 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:14.819 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T04:40:14.875 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:14.876 INFO:tasks.ceph:role: osd.5 2024-09-16T04:40:14.876 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:40:14.876 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:14.939 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:14.940 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:14.940 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:15.415 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-09-16T04:40:15.417 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-09-16T04:40:15.417 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T04:40:15.573 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T04:40:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:15.585 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:15.586 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-16T04:40:15.641 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:15.641 INFO:tasks.ceph:role: osd.9 2024-09-16T04:40:15.641 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:40:15.641 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:40:15.705 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:15.705 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:15.706 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:16.168 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-09-16T04:40:16.170 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-09-16T04:40:16.170 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-16T04:40:16.315 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-16T04:40:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:16.328 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:16.328 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-16T04:40:16.383 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:16.383 INFO:tasks.ceph:role: osd.13 2024-09-16T04:40:16.383 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:40:16.383 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:40:16.447 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:16.447 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:16.447 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:16.447 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:16.447 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:16.448 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:16.448 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:16.448 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:16.448 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:16.448 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:16.874 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-09-16T04:40:16.877 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-09-16T04:40:16.877 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-16T04:40:16.964 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-16T04:40:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:16.974 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:16.974 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:17.086 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:17.082+0000 7f6ca3a2f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T04:40:17.089 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:17.086+0000 7f6ca3a2f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T04:40:17.089 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:17.086+0000 7f6ca3a2f7c0 -1 bdev(0x55d6d3c85800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:17.089 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:17.086+0000 7f6ca3a2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T04:40:20.230 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T04:40:20.244 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:20.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:20.322+0000 7f8d636bc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T04:40:20.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:20.322+0000 7f8d636bc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T04:40:20.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:20.322+0000 7f8d636bc7c0 -1 bdev(0x5622b397f800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:20.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:20.322+0000 7f8d636bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T04:40:23.459 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T04:40:23.515 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:23.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:23.594+0000 7f649cde77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-16T04:40:23.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:23.598+0000 7f649cde77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-16T04:40:23.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:23.598+0000 7f649cde77c0 -1 bdev(0x55db53819800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:23.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:23.598+0000 7f649cde77c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-16T04:40:26.752 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-16T04:40:26.807 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:26.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:26.886+0000 7f4203b7a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-16T04:40:26.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:26.890+0000 7f4203b7a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-16T04:40:26.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:26.890+0000 7f4203b7a7c0 -1 bdev(0x564470a53800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:26.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-09-16T04:40:26.890+0000 7f4203b7a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-16T04:40:30.022 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-16T04:40:30.076 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T04:40:30.090 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:30.090 INFO:tasks.ceph:role: osd.2 2024-09-16T04:40:30.090 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:40:30.090 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:40:30.151 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:30.151 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:30.151 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:30.152 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:30.152 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:30.152 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:30.152 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:30.152 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:30.152 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:30.152 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:30.615 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-16T04:40:30.617 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-16T04:40:30.618 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-16T04:40:30.718 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T04:40:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:30.728 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:30.729 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T04:40:30.782 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:30.782 INFO:tasks.ceph:role: osd.6 2024-09-16T04:40:30.782 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:40:30.782 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:40:30.843 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:30.844 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:31.313 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-16T04:40:31.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-16T04:40:31.315 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-16T04:40:31.469 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T04:40:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:31.482 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:31.483 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-16T04:40:31.536 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:31.536 INFO:tasks.ceph:role: osd.10 2024-09-16T04:40:31.536 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:40:31.536 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:40:31.599 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:31.600 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:32.082 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-16T04:40:32.085 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-16T04:40:32.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-16T04:40:32.232 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-16T04:40:32.244 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:32.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-16T04:40:32.300 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:32.301 INFO:tasks.ceph:role: osd.14 2024-09-16T04:40:32.301 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:40:32.301 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:40:32.363 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:32.363 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:32.363 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:32.363 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:32.363 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:32.364 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:32.364 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:32.364 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:32.364 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:32.364 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:32.806 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-16T04:40:32.808 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-16T04:40:32.808 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-16T04:40:32.906 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-16T04:40:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:32.917 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:32.917 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:33.029 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:33.028+0000 7f0372ef87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:33.028+0000 7f0372ef87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:33.028+0000 7f0372ef87c0 -1 bdev(0x56079a681800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:33.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:33.028+0000 7f0372ef87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T04:40:36.166 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T04:40:36.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:36.263 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:36.260+0000 7f908f6b87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T04:40:36.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:36.264+0000 7f908f6b87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T04:40:36.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:36.264+0000 7f908f6b87c0 -1 bdev(0x561f89fed800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:36.264 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:36.264+0000 7f908f6b87c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T04:40:39.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T04:40:39.444 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:39.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:39.524+0000 7f5b751be7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-16T04:40:39.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:39.524+0000 7f5b751be7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-16T04:40:39.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:39.524+0000 7f5b751be7c0 -1 bdev(0x55dfd1327800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:39.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:39.524+0000 7f5b751be7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-16T04:40:42.664 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-16T04:40:42.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:42.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:42.800+0000 7f60574047c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-16T04:40:42.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:42.804+0000 7f60574047c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-16T04:40:42.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:42.804+0000 7f60574047c0 -1 bdev(0x55a83c345800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:42.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-16T04:40:42.804+0000 7f60574047c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-16T04:40:45.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-16T04:40:46.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T04:40:46.021 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:46.021 INFO:tasks.ceph:role: osd.3 2024-09-16T04:40:46.021 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:40:46.022 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:40:46.082 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:46.082 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:46.083 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:46.083 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:46.083 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:46.566 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-16T04:40:46.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-16T04:40:46.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T04:40:46.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T04:40:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:46.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:46.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T04:40:46.794 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:46.794 INFO:tasks.ceph:role: osd.7 2024-09-16T04:40:46.794 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:40:46.794 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:46.859 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:46.860 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:47.310 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-16T04:40:47.312 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-16T04:40:47.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-16T04:40:47.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T04:40:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:47.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:47.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-16T04:40:47.468 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:47.468 INFO:tasks.ceph:role: osd.11 2024-09-16T04:40:47.468 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:40:47.468 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:40:47.530 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:47.530 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:47.531 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:47.995 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-16T04:40:47.996 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-16T04:40:47.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-16T04:40:48.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-16T04:40:48.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:48.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-16T04:40:48.182 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T04:40:48.182 INFO:tasks.ceph:role: osd.15 2024-09-16T04:40:48.182 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:40:48.183 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:40:48.247 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:40:48.247 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:40:48.247 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:40:48.247 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T04:40:48.247 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:40:48.248 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:40:48.248 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:40:48.248 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T04:40:48.248 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:40:48.248 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:40:48.740 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-16T04:40:48.742 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-16T04:40:48.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-16T04:40:48.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-16T04:40:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:40:48.896 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T04:40:48.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:49.009 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:49.004+0000 7f0ba150f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T04:40:49.013 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:49.008+0000 7f0ba150f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T04:40:49.013 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:49.008+0000 7f0ba150f7c0 -1 bdev(0x5615f4ce5800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:49.013 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:49.008+0000 7f0ba150f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T04:40:52.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T04:40:52.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:52.249 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:52.244+0000 7f9ad76f37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T04:40:52.250 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:52.244+0000 7f9ad76f37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T04:40:52.250 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:52.244+0000 7f9ad76f37c0 -1 bdev(0x557a172a5800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:52.250 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:52.248+0000 7f9ad76f37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T04:40:55.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T04:40:55.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:55.565 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:55.560+0000 7f5a24d827c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-16T04:40:55.565 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:55.560+0000 7f5a24d827c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-16T04:40:55.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:55.560+0000 7f5a24d827c0 -1 bdev(0x55a09067d800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:55.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:55.560+0000 7f5a24d827c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-16T04:40:58.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-16T04:40:58.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:40:58.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:58.856+0000 7f5e9d14b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-16T04:40:58.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:58.856+0000 7f5e9d14b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-16T04:40:58.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:58.856+0000 7f5e9d14b7c0 -1 bdev(0x562114291800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-16T04:40:58.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-16T04:40:58.856+0000 7f5e9d14b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-16T04:41:01.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-16T04:41:02.051 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T04:41:02.051 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:02.051 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T04:41:02.066 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:02.066 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T04:41:02.120 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:02.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T04:41:02.176 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:02.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-16T04:41:02.232 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:02.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-16T04:41:02.288 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:02.288 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T04:41:02.302 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:02.303 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T04:41:02.359 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:02.359 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-16T04:41:02.414 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:02.414 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-16T04:41:02.470 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:02.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T04:41:02.485 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:02.485 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T04:41:02.540 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:02.540 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-16T04:41:02.596 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:02.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-16T04:41:02.651 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:02.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T04:41:02.664 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:02.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T04:41:02.718 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:02.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T04:41:02.773 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:02.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-16T04:41:02.825 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:02.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-16T04:41:02.881 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:02.882 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T04:41:02.930 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T04:41:02.931 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:41:02.933 DEBUG:teuthology.orchestra.run.smithi139:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:41:02.935 DEBUG:teuthology.orchestra.run.smithi144:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.y] 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCdtudmY+P6OBAAjGWZLy14okQeLIVvPGo7wA== 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQChtudmK9/mABAAVMD1hkyc4dnG9CBXjkekeg== 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.4] 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCktudmxsiMDxAAZH6vPwKHo04jCg+GC6zYgA== 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.8] 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCntudmwtMsJBAAS65UL4zJZ+WdgXhK/CRC3w== 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.12] 2024-09-16T04:41:02.943 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCqtudm1d0iNxAAZz1pvMNgnGu/npq7xU9Fig== 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCxtudmpPIlBRAAZQmHKAgzkhLQHGbMp7wSNg== 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.5] 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC0tudmJPFoExAAjqo2PTMfn0I12KCn8tO32w== 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.9] 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC3tudmsqexIxAAE3yznUxXu8isxU31y+5ODg== 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.13] 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC6tudmfP4PNRAAWt6yJjP7pHOSNrqN2bWRJA== 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDBtudmSADHARAAz1gKYqWLZXwpauJ95k5jiA== 2024-09-16T04:41:02.944 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.6] 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDEtudm977NDxAABo5fBxs8xUTCyXOJ17IHWQ== 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.10] 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDHtudmRuGBHxAAmegpCREM6j3/8SVlCn3BNQ== 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.14] 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDKtudmK8/5LxAA8kt08fXswbEHPjHFzB4PjA== 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCetudmG70DABAAp3u7zrnSL9e9d9kmKOmdeA== 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDRtudmKmCGABAArasqy9PiMF3FmyUtMVhW6w== 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.7] 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDUtudmQDvUDhAAhuXtGFIBV1eHiLTfMsHX1A== 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.11] 2024-09-16T04:41:02.945 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDXtudm+h6mIRAAfJDpUyTt3MD3oe1dhGdXyg== 2024-09-16T04:41:02.946 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.15] 2024-09-16T04:41:02.946 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDatudmObJJMxAAFpIqWJ826wbsBKSfpicRuw== 2024-09-16T04:41:02.946 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-09-16T04:41:02.946 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCetudmBOw+BRAAC9vYGL8vPRGX20u5hO7C/Q== 2024-09-16T04:41:02.947 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.y] 2024-09-16T04:41:02.947 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCdtudmY+P6OBAAjGWZLy14okQeLIVvPGo7wA== 2024-09-16T04:41:02.947 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2024-09-16T04:41:02.947 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQChtudmK9/mABAAVMD1hkyc4dnG9CBXjkekeg== 2024-09-16T04:41:02.947 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.4] 2024-09-16T04:41:02.947 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCktudmxsiMDxAAZH6vPwKHo04jCg+GC6zYgA== 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.8] 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCntudmwtMsJBAAS65UL4zJZ+WdgXhK/CRC3w== 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.12] 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCqtudm1d0iNxAAZz1pvMNgnGu/npq7xU9Fig== 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCxtudmpPIlBRAAZQmHKAgzkhLQHGbMp7wSNg== 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.5] 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQC0tudmJPFoExAAjqo2PTMfn0I12KCn8tO32w== 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.9] 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQC3tudmsqexIxAAE3yznUxXu8isxU31y+5ODg== 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.13] 2024-09-16T04:41:02.948 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQC6tudmfP4PNRAAWt6yJjP7pHOSNrqN2bWRJA== 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDBtudmSADHARAAz1gKYqWLZXwpauJ95k5jiA== 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.6] 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDEtudm977NDxAABo5fBxs8xUTCyXOJ17IHWQ== 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.10] 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDHtudmRuGBHxAAmegpCREM6j3/8SVlCn3BNQ== 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.14] 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDKtudmK8/5LxAA8kt08fXswbEHPjHFzB4PjA== 2024-09-16T04:41:02.949 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.x] 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCetudmG70DABAAp3u7zrnSL9e9d9kmKOmdeA== 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.3] 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDRtudmKmCGABAArasqy9PiMF3FmyUtMVhW6w== 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.7] 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDUtudmQDvUDhAAhuXtGFIBV1eHiLTfMsHX1A== 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.11] 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDXtudm+h6mIRAAfJDpUyTt3MD3oe1dhGdXyg== 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.15] 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDatudmObJJMxAAFpIqWJ826wbsBKSfpicRuw== 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2024-09-16T04:41:02.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCetudmBOw+BRAAC9vYGL8vPRGX20u5hO7C/Q== 2024-09-16T04:41:02.951 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.y] 2024-09-16T04:41:02.951 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCdtudmY+P6OBAAjGWZLy14okQeLIVvPGo7wA== 2024-09-16T04:41:02.951 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2024-09-16T04:41:02.951 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQChtudmK9/mABAAVMD1hkyc4dnG9CBXjkekeg== 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.4] 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCktudmxsiMDxAAZH6vPwKHo04jCg+GC6zYgA== 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.8] 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCntudmwtMsJBAAS65UL4zJZ+WdgXhK/CRC3w== 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.12] 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCqtudm1d0iNxAAZz1pvMNgnGu/npq7xU9Fig== 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCxtudmpPIlBRAAZQmHKAgzkhLQHGbMp7wSNg== 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.5] 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC0tudmJPFoExAAjqo2PTMfn0I12KCn8tO32w== 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.9] 2024-09-16T04:41:02.952 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC3tudmsqexIxAAE3yznUxXu8isxU31y+5ODg== 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.13] 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQC6tudmfP4PNRAAWt6yJjP7pHOSNrqN2bWRJA== 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDBtudmSADHARAAz1gKYqWLZXwpauJ95k5jiA== 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.6] 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDEtudm977NDxAABo5fBxs8xUTCyXOJ17IHWQ== 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.10] 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDHtudmRuGBHxAAmegpCREM6j3/8SVlCn3BNQ== 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.14] 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDKtudmK8/5LxAA8kt08fXswbEHPjHFzB4PjA== 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCetudmG70DABAAp3u7zrnSL9e9d9kmKOmdeA== 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.3] 2024-09-16T04:41:02.953 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDRtudmKmCGABAArasqy9PiMF3FmyUtMVhW6w== 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.7] 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDUtudmQDvUDhAAhuXtGFIBV1eHiLTfMsHX1A== 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.11] 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDXtudm+h6mIRAAfJDpUyTt3MD3oe1dhGdXyg== 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.15] 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQDatudmObJJMxAAFpIqWJ826wbsBKSfpicRuw== 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2024-09-16T04:41:02.954 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCetudmBOw+BRAAC9vYGL8vPRGX20u5hO7C/Q== 2024-09-16T04:41:02.954 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:41:02.992 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:41:02.995 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:41:03.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.078 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.125 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.169 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.171 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.215 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.257 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.345 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.348 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.429 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.513 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.601 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.660 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.685 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.700 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.745 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.769 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.783 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.827 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.864 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.906 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.909 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.944 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:03.993 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:41:04.077 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:41:04.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:41:04.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.161 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.187 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.230 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.245 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.267 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.310 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.338 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.347 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.425 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:41:04.479 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:41:04.481 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:41:04.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:41:04.566 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T04:41:04.566 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T04:41:04.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:41:04.676 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T04:41:04.687 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T04:41:04.700 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:41:04.793 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T04:41:04.805 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T04:41:04.820 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:41:04.913 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T04:41:04.925 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:41:04.928 DEBUG:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:41:04.930 DEBUG:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:41:04.971 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T04:41:04.972 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:41:04.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:41:04.974 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:41:04.974 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:41:04.975 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:41:04.977 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:41:04.977 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:41:04.977 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:41:05.017 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:41:05.017 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T04:41:05.017 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T04:41:05.017 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T04:41:05.020 INFO:tasks.ceph.mgr.y:Started 2024-09-16T04:41:05.020 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T04:41:05.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T04:41:05.023 INFO:tasks.ceph.mgr.x:Started 2024-09-16T04:41:05.023 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T04:41:05.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T04:41:10.508 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-09-16T04:41:10.528 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T04:41:10.528 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T04:41:10.528 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:10.528 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T04:41:10.540 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:10.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T04:41:10.595 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:10.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-16T04:41:10.601 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:10.596+0000 7f57607f4180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:41:10.652 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-16T04:41:10.652 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-16T04:41:10.680 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:10.678+0000 7f887cbb6180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:41:10.692 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:10.692 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T04:41:10.707 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:10.707 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T04:41:10.762 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:10.763 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-16T04:41:10.821 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-16T04:41:10.821 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-16T04:41:10.874 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:10.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T04:41:10.888 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:10.889 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T04:41:10.942 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:10.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-16T04:41:10.997 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-16T04:41:10.997 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-16T04:41:11.051 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:11.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T04:41:11.067 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:11.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T04:41:11.088 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:11.084+0000 7f57607f4180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:41:11.100 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:11.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-16T04:41:11.153 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T04:41:11.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-16T04:41:11.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new b9529dc0-e1eb-4b2f-8297-45254ddbffa8 0 2024-09-16T04:41:11.212 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:11.210+0000 7f887cbb6180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:41:11.286 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:11.280+0000 7f57607f4180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:41:11.439 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:11.434+0000 7f887cbb6180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:41:11.447 INFO:teuthology.orchestra.run.smithi165.stdout:0 2024-09-16T04:41:11.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new dceca258-f5a1-4e8d-be6e-9d38bc5f14e1 1 2024-09-16T04:41:11.801 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-09-16T04:41:11.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 6bfc934a-0ca2-4b1f-9e09-14c18a7d5116 2 2024-09-16T04:41:11.844 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:11.840+0000 7f57607f4180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:41:11.899 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:11.896+0000 7f57607f4180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:41:11.954 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:11.952+0000 7f57607f4180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:41:12.022 INFO:teuthology.orchestra.run.smithi165.stdout:2 2024-09-16T04:41:12.032 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:12.028+0000 7f57607f4180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:41:12.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 0b949ef9-82e3-483f-8be6-65e04d169c4e 3 2024-09-16T04:41:12.040 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.038+0000 7f887cbb6180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:41:12.085 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:12.080+0000 7f57607f4180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:41:12.096 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.094+0000 7f887cbb6180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:41:12.151 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.146+0000 7f887cbb6180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:41:12.164 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:12.160+0000 7f57607f4180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:41:12.218 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:12.216+0000 7f57607f4180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:41:12.233 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.230+0000 7f887cbb6180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:41:12.282 INFO:teuthology.orchestra.run.smithi165.stdout:3 2024-09-16T04:41:12.288 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.286+0000 7f887cbb6180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:41:12.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new d35e5791-7bb4-493d-99ec-3ff942117371 4 2024-09-16T04:41:12.369 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.366+0000 7f887cbb6180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:41:12.372 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:12.368+0000 7f57607f4180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:41:12.425 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.422+0000 7f887cbb6180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:41:12.435 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:12.432+0000 7f57607f4180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:41:12.548 INFO:teuthology.orchestra.run.smithi165.stdout:4 2024-09-16T04:41:12.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 159725f2-3713-4f0e-9e44-55b678bf09fa 5 2024-09-16T04:41:12.584 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.582+0000 7f887cbb6180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:41:12.649 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:12.646+0000 7f887cbb6180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:41:12.859 INFO:teuthology.orchestra.run.smithi165.stdout:5 2024-09-16T04:41:12.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new e642cfb3-cf94-4124-93db-9e2bdf1b7c5f 6 2024-09-16T04:41:13.066 INFO:teuthology.orchestra.run.smithi165.stdout:6 2024-09-16T04:41:13.079 INFO:tasks.ceph.mgr.x.smithi165.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T04:41:13.079 INFO:tasks.ceph.mgr.x.smithi165.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T04:41:13.079 INFO:tasks.ceph.mgr.x.smithi165.stderr: from numpy import show_config as show_numpy_config 2024-09-16T04:41:13.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb 7 2024-09-16T04:41:13.132 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:13.128+0000 7f57607f4180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:41:13.272 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:13.268+0000 7f57607f4180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:41:13.277 INFO:teuthology.orchestra.run.smithi165.stdout:7 2024-09-16T04:41:13.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 04681c69-387b-4dd8-9eb4-687f34ba9dab 8 2024-09-16T04:41:13.369 INFO:tasks.ceph.mgr.y.smithi059.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T04:41:13.369 INFO:tasks.ceph.mgr.y.smithi059.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T04:41:13.369 INFO:tasks.ceph.mgr.y.smithi059.stderr: from numpy import show_config as show_numpy_config 2024-09-16T04:41:13.430 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:13.426+0000 7f887cbb6180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:41:13.432 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:13.428+0000 7f57607f4180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:41:13.487 INFO:teuthology.orchestra.run.smithi165.stdout:8 2024-09-16T04:41:13.496 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:13.492+0000 7f57607f4180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:41:13.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 713a3b76-8daf-4e44-9d0b-68369a9da692 9 2024-09-16T04:41:13.621 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:13.618+0000 7f887cbb6180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:41:13.637 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:13.632+0000 7f57607f4180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:41:13.692 INFO:teuthology.orchestra.run.smithi165.stdout:9 2024-09-16T04:41:13.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 81450492-3178-4643-8066-847fcdd266b4 10 2024-09-16T04:41:13.796 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:13.790+0000 7f887cbb6180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:41:13.869 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:13.866+0000 7f887cbb6180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:41:13.894 INFO:teuthology.orchestra.run.smithi165.stdout:10 2024-09-16T04:41:13.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 9938b272-43e2-4b48-91ca-42cbe5f26664 11 2024-09-16T04:41:14.020 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:14.018+0000 7f887cbb6180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:41:14.110 INFO:teuthology.orchestra.run.smithi165.stdout:11 2024-09-16T04:41:14.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new cd20410e-c17f-43d7-8512-9efd8193a251 12 2024-09-16T04:41:14.251 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:14.248+0000 7f57607f4180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:41:14.323 INFO:teuthology.orchestra.run.smithi165.stdout:12 2024-09-16T04:41:14.327 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:14.324+0000 7f57607f4180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:41:14.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 953622d2-9e72-451d-8863-d8c55139c65b 13 2024-09-16T04:41:14.515 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:14.512+0000 7f57607f4180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:41:14.537 INFO:teuthology.orchestra.run.smithi165.stdout:13 2024-09-16T04:41:14.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new e6388924-0444-47c0-b61c-7ca66661237f 14 2024-09-16T04:41:14.592 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:14.588+0000 7f57607f4180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:41:14.676 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:14.670+0000 7f887cbb6180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:41:14.745 INFO:teuthology.orchestra.run.smithi165.stdout:14 2024-09-16T04:41:14.755 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:14.750+0000 7f887cbb6180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:41:14.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 8335941d-9563-4527-90f6-3b7f9129f530 15 2024-09-16T04:41:14.892 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:14.888+0000 7f57607f4180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:41:14.947 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:14.942+0000 7f887cbb6180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:41:14.953 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:14.948+0000 7f57607f4180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:41:15.005 INFO:teuthology.orchestra.run.smithi165.stdout:15 2024-09-16T04:41:15.020 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T04:41:15.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T04:41:15.023 INFO:tasks.ceph.osd.0:Started 2024-09-16T04:41:15.023 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T04:41:15.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T04:41:15.025 INFO:tasks.ceph.osd.4:Started 2024-09-16T04:41:15.025 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-16T04:41:15.025 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-16T04:41:15.027 INFO:tasks.ceph.osd.8:Started 2024-09-16T04:41:15.027 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-16T04:41:15.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-16T04:41:15.029 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:15.026+0000 7f887cbb6180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:41:15.029 INFO:tasks.ceph.osd.12:Started 2024-09-16T04:41:15.029 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:41:15.030 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:41:15.033 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:41:15.034 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T04:41:15.034 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T04:41:15.037 INFO:tasks.ceph.osd.5:Started 2024-09-16T04:41:15.037 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-16T04:41:15.037 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-16T04:41:15.040 INFO:tasks.ceph.osd.9:Started 2024-09-16T04:41:15.041 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-16T04:41:15.041 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-16T04:41:15.044 INFO:tasks.ceph.osd.13:Started 2024-09-16T04:41:15.044 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T04:41:15.044 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T04:41:15.046 INFO:tasks.ceph.osd.2:Started 2024-09-16T04:41:15.047 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T04:41:15.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T04:41:15.050 INFO:tasks.ceph.osd.6:Started 2024-09-16T04:41:15.050 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-16T04:41:15.050 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-16T04:41:15.053 INFO:tasks.ceph.osd.10:Started 2024-09-16T04:41:15.053 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-16T04:41:15.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-16T04:41:15.056 INFO:tasks.ceph.osd.14:Started 2024-09-16T04:41:15.056 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T04:41:15.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T04:41:15.058 INFO:tasks.ceph.osd.3:Started 2024-09-16T04:41:15.059 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T04:41:15.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T04:41:15.061 INFO:tasks.ceph.osd.7:Started 2024-09-16T04:41:15.061 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-16T04:41:15.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-16T04:41:15.063 INFO:tasks.ceph.osd.11:Started 2024-09-16T04:41:15.063 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-16T04:41:15.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-16T04:41:15.065 INFO:tasks.ceph.osd.15:Started 2024-09-16T04:41:15.065 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:41:15.251 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:15.251 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":18,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:41:15.004474+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:41:15.004474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:41:15.267 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T04:41:15.268 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T04:41:15.367 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:15.362+0000 7f887cbb6180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:41:15.434 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-09-16T04:41:15.430+0000 7f887cbb6180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:41:16.025 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:16.020+0000 7f5707328640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T04:41:16.025 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-16T04:41:16.020+0000 7f5707328640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T04:41:16.167 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:16.164+0000 7f2571b827c0 -1 Falling back to public interface 2024-09-16T04:41:16.181 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:16.179+0000 7f6df2fea7c0 -1 Falling back to public interface 2024-09-16T04:41:16.199 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:16.196+0000 7f499eed57c0 -1 Falling back to public interface 2024-09-16T04:41:16.208 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:16.207+0000 7f64974247c0 -1 Falling back to public interface 2024-09-16T04:41:16.209 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:16.207+0000 7fef735d47c0 -1 Falling back to public interface 2024-09-16T04:41:16.209 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:16.206+0000 7f570791d7c0 -1 Falling back to public interface 2024-09-16T04:41:16.268 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:16.266+0000 7f902192e7c0 -1 Falling back to public interface 2024-09-16T04:41:16.296 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:16.295+0000 7f7377c527c0 -1 Falling back to public interface 2024-09-16T04:41:16.303 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:16.302+0000 7fa7769ed7c0 -1 Falling back to public interface 2024-09-16T04:41:16.361 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:16.358+0000 7fe034bbf7c0 -1 Falling back to public interface 2024-09-16T04:41:16.429 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:16.426+0000 7f2a314f57c0 -1 Falling back to public interface 2024-09-16T04:41:16.480 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:16.478+0000 7ffad79767c0 -1 Falling back to public interface 2024-09-16T04:41:16.520 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:16.518+0000 7f25be0dd7c0 -1 Falling back to public interface 2024-09-16T04:41:16.636 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:16.632+0000 7f2e081507c0 -1 Falling back to public interface 2024-09-16T04:41:16.706 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:16.650+0000 7ff1f1eac7c0 -1 Falling back to public interface 2024-09-16T04:41:16.907 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:41:16.944 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:16.940+0000 7f097853c7c0 -1 Falling back to public interface 2024-09-16T04:41:17.214 INFO:teuthology.misc.health.smithi059.stdout: 2024-09-16T04:41:17.214 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":18,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:41:15.004474+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:41:15.004474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:41:17.231 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-16T04:41:20.182 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:20.178+0000 7f570791d7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T04:41:20.210 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:20.208+0000 7f499eed57c0 -1 osd.11 0 log_to_monitors true 2024-09-16T04:41:20.234 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:20.231+0000 7f6df2fea7c0 -1 osd.6 0 log_to_monitors true 2024-09-16T04:41:20.263 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:20.259+0000 7f64974247c0 -1 osd.10 0 log_to_monitors true 2024-09-16T04:41:20.299 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:20.294+0000 7fe034bbf7c0 -1 osd.12 0 log_to_monitors true 2024-09-16T04:41:20.322 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:20.320+0000 7f2571b827c0 -1 osd.3 0 log_to_monitors true 2024-09-16T04:41:20.340 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:20.337+0000 7fa7769ed7c0 -1 osd.9 0 log_to_monitors true 2024-09-16T04:41:20.402 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:20.399+0000 7f7377c527c0 -1 osd.2 0 log_to_monitors true 2024-09-16T04:41:20.434 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:20.433+0000 7ffad79767c0 -1 osd.1 0 log_to_monitors true 2024-09-16T04:41:20.462 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:20.458+0000 7f2a314f57c0 -1 osd.8 0 log_to_monitors true 2024-09-16T04:41:20.532 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:20.529+0000 7f25be0dd7c0 -1 osd.13 0 log_to_monitors true 2024-09-16T04:41:20.591 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:20.589+0000 7f902192e7c0 -1 osd.5 0 log_to_monitors true 2024-09-16T04:41:20.649 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:20.644+0000 7f2e081507c0 -1 osd.7 0 log_to_monitors true 2024-09-16T04:41:20.757 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:20.754+0000 7ff1f1eac7c0 -1 osd.4 0 log_to_monitors true 2024-09-16T04:41:20.873 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:20.871+0000 7fef735d47c0 -1 osd.14 0 log_to_monitors true 2024-09-16T04:41:20.912 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:20.908+0000 7f097853c7c0 -1 osd.15 0 log_to_monitors true 2024-09-16T04:41:21.346 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:21.344+0000 7f499ae7e640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T04:41:21.359 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:21.356+0000 7f4996cbb640 -1 osd.11 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:21.988 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:21.986+0000 7f57040d8640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:41:22.017 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:22.014+0000 7f56fb6fb640 -1 osd.0 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.349 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:22.344+0000 7f09744e5640 -1 osd.15 0 waiting for initial osdmap 2024-09-16T04:41:22.350 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:22.348+0000 7f256db2b640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T04:41:22.350 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:22.346+0000 7fe030b68640 -1 osd.12 0 waiting for initial osdmap 2024-09-16T04:41:22.351 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:22.346+0000 7f2a2d49e640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T04:41:22.352 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:22.351+0000 7f6deef93640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T04:41:22.357 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:22.355+0000 7f7373bfb640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:41:22.357 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:22.353+0000 7f901d8d7640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T04:41:22.358 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:22.355+0000 7f64933cd640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T04:41:22.372 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:22.371+0000 7f648f20a640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.373 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:22.371+0000 7f6deadd0640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.377 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:22.374+0000 7fe02c9a5640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.387 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:22.385+0000 7f9019714640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.395 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:22.392+0000 7f2569968640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.472 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:22.471+0000 7f736fa38640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.474 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:22.472+0000 7f0970322640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.480 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:22.474+0000 7f2a292db640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:22.940 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:22.938+0000 7ff1ee667640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T04:41:22.941 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:22.936+0000 7f2e0490b640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T04:41:22.945 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:22.943+0000 7fef6f57d640 -1 osd.14 0 waiting for initial osdmap 2024-09-16T04:41:23.159 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:23.156+0000 7f2dfbf2e640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:23.161 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:23.158+0000 7ff1e5c8a640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:23.177 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:23.175+0000 7fef673b2640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:23.219 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:23.217+0000 7fa772996640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T04:41:23.251 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:23.249+0000 7ffad391f640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T04:41:23.285 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:23.281+0000 7f25ba086640 -1 osd.13 0 waiting for initial osdmap 2024-09-16T04:41:23.443 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:23.441+0000 7fa76a7cb640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:23.487 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:23.485+0000 7ffacb754640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:23.520 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:23.517+0000 7f25b1ebb640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:41:24.840 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:41:25.156 INFO:teuthology.misc.health.smithi059.stdout: 2024-09-16T04:41:25.157 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":23,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:41:24.359002+0000","last_up_change":"2024-09-16T04:41:24.359002+0000","last_in_change":"2024-09-16T04:41:15.004474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:41:23.074365+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6801","nonce":1228226348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6803","nonce":1228226348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6807","nonce":1228226348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6805","nonce":1228226348}]},"public_addr":"172.21.15.59:6801/1228226348","cluster_addr":"172.21.15.59:6803/1228226348","heartbeat_back_addr":"172.21.15.59:6807/1228226348","heartbeat_front_addr":"172.21.15.59:6805/1228226348","state":["exists","up"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6817","nonce":2731115256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6819","nonce":2731115256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6823","nonce":2731115256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6821","nonce":2731115256}]},"public_addr":"172.21.15.139:6817/2731115256","cluster_addr":"172.21.15.139:6819/2731115256","heartbeat_back_addr":"172.21.15.139:6823/2731115256","heartbeat_front_addr":"172.21.15.139:6821/2731115256","state":["exists","up"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6825","nonce":76637888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6827","nonce":76637888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6831","nonce":76637888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6829","nonce":76637888}]},"public_addr":"172.21.15.144:6825/76637888","cluster_addr":"172.21.15.144:6827/76637888","heartbeat_back_addr":"172.21.15.144:6831/76637888","heartbeat_front_addr":"172.21.15.144:6829/76637888","state":["exists","up"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6803","nonce":58455506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6805","nonce":58455506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6809","nonce":58455506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6807","nonce":58455506}]},"public_addr":"172.21.15.165:6803/58455506","cluster_addr":"172.21.15.165:6805/58455506","heartbeat_back_addr":"172.21.15.165:6809/58455506","heartbeat_front_addr":"172.21.15.165:6807/58455506","state":["exists","up"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6825","nonce":2712399685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6827","nonce":2712399685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6831","nonce":2712399685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6829","nonce":2712399685}]},"public_addr":"172.21.15.59:6825/2712399685","cluster_addr":"172.21.15.59:6827/2712399685","heartbeat_back_addr":"172.21.15.59:6831/2712399685","heartbeat_front_addr":"172.21.15.59:6829/2712399685","state":["exists","up"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6801","nonce":2946170755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6803","nonce":2946170755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6807","nonce":2946170755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6805","nonce":2946170755}]},"public_addr":"172.21.15.139:6801/2946170755","cluster_addr":"172.21.15.139:6803/2946170755","heartbeat_back_addr":"172.21.15.139:6807/2946170755","heartbeat_front_addr":"172.21.15.139:6805/2946170755","state":["exists","up"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6801","nonce":3812816248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6803","nonce":3812816248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6807","nonce":3812816248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6805","nonce":3812816248}]},"public_addr":"172.21.15.144:6801/3812816248","cluster_addr":"172.21.15.144:6803/3812816248","heartbeat_back_addr":"172.21.15.144:6807/3812816248","heartbeat_front_addr":"172.21.15.144:6805/3812816248","state":["exists","up"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6819","nonce":1352862134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6821","nonce":1352862134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6825","nonce":1352862134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6823","nonce":1352862134}]},"public_addr":"172.21.15.165:6819/1352862134","cluster_addr":"172.21.15.165:6821/1352862134","heartbeat_back_addr":"172.21.15.165:6825/1352862134","heartbeat_front_addr":"172.21.15.165:6823/1352862134","state":["exists","up"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6817","nonce":1451673997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6819","nonce":1451673997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6823","nonce":1451673997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6821","nonce":1451673997}]},"public_addr":"172.21.15.59:6817/1451673997","cluster_addr":"172.21.15.59:6819/1451673997","heartbeat_back_addr":"172.21.15.59:6823/1451673997","heartbeat_front_addr":"172.21.15.59:6821/1451673997","state":["exists","up"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6809","nonce":174106571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6811","nonce":174106571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6815","nonce":174106571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6813","nonce":174106571}]},"public_addr":"172.21.15.139:6809/174106571","cluster_addr":"172.21.15.139:6811/174106571","heartbeat_back_addr":"172.21.15.139:6815/174106571","heartbeat_front_addr":"172.21.15.139:6813/174106571","state":["exists","up"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6809","nonce":66779937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6812","nonce":66779937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6818","nonce":66779937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6815","nonce":66779937}]},"public_addr":"172.21.15.144:6809/66779937","cluster_addr":"172.21.15.144:6812/66779937","heartbeat_back_addr":"172.21.15.144:6818/66779937","heartbeat_front_addr":"172.21.15.144:6815/66779937","state":["exists","up"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6811","nonce":49204916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6813","nonce":49204916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6817","nonce":49204916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6815","nonce":49204916}]},"public_addr":"172.21.15.165:6811/49204916","cluster_addr":"172.21.15.165:6813/49204916","heartbeat_back_addr":"172.21.15.165:6817/49204916","heartbeat_front_addr":"172.21.15.165:6815/49204916","state":["exists","up"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6809","nonce":3097346146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6811","nonce":3097346146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6815","nonce":3097346146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6813","nonce":3097346146}]},"public_addr":"172.21.15.59:6809/3097346146","cluster_addr":"172.21.15.59:6811/3097346146","heartbeat_back_addr":"172.21.15.59:6815/3097346146","heartbeat_front_addr":"172.21.15.59:6813/3097346146","state":["exists","up"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6825","nonce":2903817823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6827","nonce":2903817823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6831","nonce":2903817823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6829","nonce":2903817823}]},"public_addr":"172.21.15.139:6825/2903817823","cluster_addr":"172.21.15.139:6827/2903817823","heartbeat_back_addr":"172.21.15.139:6831/2903817823","heartbeat_front_addr":"172.21.15.139:6829/2903817823","state":["exists","up"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6811","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6814","nonce":229023981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6819","nonce":229023981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6823","nonce":229023981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6821","nonce":229023981}]},"public_addr":"172.21.15.144:6814/229023981","cluster_addr":"172.21.15.144:6819/229023981","heartbeat_back_addr":"172.21.15.144:6823/229023981","heartbeat_front_addr":"172.21.15.144:6821/229023981","state":["exists","up"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6827","nonce":2758676961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6829","nonce":2758676961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6833","nonce":2758676961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6831","nonce":2758676961}]},"public_addr":"172.21.15.165:6827/2758676961","cluster_addr":"172.21.15.165:6829/2758676961","heartbeat_back_addr":"172.21.15.165:6833/2758676961","heartbeat_front_addr":"172.21.15.165:6831/2758676961","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.223183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.389820+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.279164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.793764+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.571561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.265206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.684523+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.452789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.258982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.166555+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.256846+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.852615+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.867981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:41:25.174 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-16T04:41:25.174 INFO:tasks.ceph:Creating RBD pool 2024-09-16T04:41:25.174 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T04:41:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'rbd' created 2024-09-16T04:41:26.403 DEBUG:teuthology.orchestra.run.smithi059:> rbd --cluster ceph pool init rbd 2024-09-16T04:41:29.643 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T04:41:29.643 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T04:41:29.643 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T04:41:29.644 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T04:41:29.644 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T04:41:30.014 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:30.032 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":1021693637},{"type":"v1","addr":"172.21.15.165:6801","nonce":1021693637}]},"active_addr":"172.21.15.165:6801/1021693637","active_change":"2024-09-16T04:41:14.977343+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":1070274904}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":2217103098}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":2478145654}]}]} 2024-09-16T04:41:30.040 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T04:41:30.040 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:41:30.040 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:41:30.349 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:30.349 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:41:29.403827+0000","last_up_change":"2024-09-16T04:41:24.359002+0000","last_in_change":"2024-09-16T04:41:15.004474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:41:23.074365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:41:25.522114+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6801","nonce":1228226348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6803","nonce":1228226348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6807","nonce":1228226348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6805","nonce":1228226348}]},"public_addr":"172.21.15.59:6801/1228226348","cluster_addr":"172.21.15.59:6803/1228226348","heartbeat_back_addr":"172.21.15.59:6807/1228226348","heartbeat_front_addr":"172.21.15.59:6805/1228226348","state":["exists","up"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6817","nonce":2731115256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6819","nonce":2731115256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6823","nonce":2731115256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6821","nonce":2731115256}]},"public_addr":"172.21.15.139:6817/2731115256","cluster_addr":"172.21.15.139:6819/2731115256","heartbeat_back_addr":"172.21.15.139:6823/2731115256","heartbeat_front_addr":"172.21.15.139:6821/2731115256","state":["exists","up"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6825","nonce":76637888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6827","nonce":76637888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6831","nonce":76637888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6829","nonce":76637888}]},"public_addr":"172.21.15.144:6825/76637888","cluster_addr":"172.21.15.144:6827/76637888","heartbeat_back_addr":"172.21.15.144:6831/76637888","heartbeat_front_addr":"172.21.15.144:6829/76637888","state":["exists","up"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6803","nonce":58455506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6805","nonce":58455506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6809","nonce":58455506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6807","nonce":58455506}]},"public_addr":"172.21.15.165:6803/58455506","cluster_addr":"172.21.15.165:6805/58455506","heartbeat_back_addr":"172.21.15.165:6809/58455506","heartbeat_front_addr":"172.21.15.165:6807/58455506","state":["exists","up"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6825","nonce":2712399685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6827","nonce":2712399685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6831","nonce":2712399685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6829","nonce":2712399685}]},"public_addr":"172.21.15.59:6825/2712399685","cluster_addr":"172.21.15.59:6827/2712399685","heartbeat_back_addr":"172.21.15.59:6831/2712399685","heartbeat_front_addr":"172.21.15.59:6829/2712399685","state":["exists","up"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6801","nonce":2946170755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6803","nonce":2946170755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6807","nonce":2946170755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6805","nonce":2946170755}]},"public_addr":"172.21.15.139:6801/2946170755","cluster_addr":"172.21.15.139:6803/2946170755","heartbeat_back_addr":"172.21.15.139:6807/2946170755","heartbeat_front_addr":"172.21.15.139:6805/2946170755","state":["exists","up"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6801","nonce":3812816248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6803","nonce":3812816248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6807","nonce":3812816248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6805","nonce":3812816248}]},"public_addr":"172.21.15.144:6801/3812816248","cluster_addr":"172.21.15.144:6803/3812816248","heartbeat_back_addr":"172.21.15.144:6807/3812816248","heartbeat_front_addr":"172.21.15.144:6805/3812816248","state":["exists","up"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6819","nonce":1352862134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6821","nonce":1352862134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6825","nonce":1352862134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6823","nonce":1352862134}]},"public_addr":"172.21.15.165:6819/1352862134","cluster_addr":"172.21.15.165:6821/1352862134","heartbeat_back_addr":"172.21.15.165:6825/1352862134","heartbeat_front_addr":"172.21.15.165:6823/1352862134","state":["exists","up"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6817","nonce":1451673997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6819","nonce":1451673997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6823","nonce":1451673997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6821","nonce":1451673997}]},"public_addr":"172.21.15.59:6817/1451673997","cluster_addr":"172.21.15.59:6819/1451673997","heartbeat_back_addr":"172.21.15.59:6823/1451673997","heartbeat_front_addr":"172.21.15.59:6821/1451673997","state":["exists","up"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6809","nonce":174106571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6811","nonce":174106571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6815","nonce":174106571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6813","nonce":174106571}]},"public_addr":"172.21.15.139:6809/174106571","cluster_addr":"172.21.15.139:6811/174106571","heartbeat_back_addr":"172.21.15.139:6815/174106571","heartbeat_front_addr":"172.21.15.139:6813/174106571","state":["exists","up"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6809","nonce":66779937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6812","nonce":66779937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6818","nonce":66779937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6815","nonce":66779937}]},"public_addr":"172.21.15.144:6809/66779937","cluster_addr":"172.21.15.144:6812/66779937","heartbeat_back_addr":"172.21.15.144:6818/66779937","heartbeat_front_addr":"172.21.15.144:6815/66779937","state":["exists","up"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6811","nonce":49204916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6813","nonce":49204916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6817","nonce":49204916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6815","nonce":49204916}]},"public_addr":"172.21.15.165:6811/49204916","cluster_addr":"172.21.15.165:6813/49204916","heartbeat_back_addr":"172.21.15.165:6817/49204916","heartbeat_front_addr":"172.21.15.165:6815/49204916","state":["exists","up"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6809","nonce":3097346146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6811","nonce":3097346146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6815","nonce":3097346146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6813","nonce":3097346146}]},"public_addr":"172.21.15.59:6809/3097346146","cluster_addr":"172.21.15.59:6811/3097346146","heartbeat_back_addr":"172.21.15.59:6815/3097346146","heartbeat_front_addr":"172.21.15.59:6813/3097346146","state":["exists","up"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6825","nonce":2903817823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6827","nonce":2903817823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6831","nonce":2903817823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6829","nonce":2903817823}]},"public_addr":"172.21.15.139:6825/2903817823","cluster_addr":"172.21.15.139:6827/2903817823","heartbeat_back_addr":"172.21.15.139:6831/2903817823","heartbeat_front_addr":"172.21.15.139:6829/2903817823","state":["exists","up"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6811","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6814","nonce":229023981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6819","nonce":229023981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6823","nonce":229023981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6821","nonce":229023981}]},"public_addr":"172.21.15.144:6814/229023981","cluster_addr":"172.21.15.144:6819/229023981","heartbeat_back_addr":"172.21.15.144:6823/229023981","heartbeat_front_addr":"172.21.15.144:6821/229023981","state":["exists","up"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6827","nonce":2758676961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6829","nonce":2758676961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6833","nonce":2758676961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6831","nonce":2758676961}]},"public_addr":"172.21.15.165:6827/2758676961","cluster_addr":"172.21.15.165:6829/2758676961","heartbeat_back_addr":"172.21.15.165:6833/2758676961","heartbeat_front_addr":"172.21.15.165:6831/2758676961","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.223183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.481924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.389820+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.279164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.793764+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.571561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.265206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.684523+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.452789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.332716+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.258982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.166555+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.256846+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.572294+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.852615+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.867981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:41:30.367 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:41:30.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:41:30.672 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:30.672 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:41:29.403827+0000","last_up_change":"2024-09-16T04:41:24.359002+0000","last_in_change":"2024-09-16T04:41:15.004474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:41:23.074365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:41:25.522114+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6801","nonce":1228226348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6803","nonce":1228226348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6807","nonce":1228226348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6805","nonce":1228226348}]},"public_addr":"172.21.15.59:6801/1228226348","cluster_addr":"172.21.15.59:6803/1228226348","heartbeat_back_addr":"172.21.15.59:6807/1228226348","heartbeat_front_addr":"172.21.15.59:6805/1228226348","state":["exists","up"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6817","nonce":2731115256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6819","nonce":2731115256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6823","nonce":2731115256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6821","nonce":2731115256}]},"public_addr":"172.21.15.139:6817/2731115256","cluster_addr":"172.21.15.139:6819/2731115256","heartbeat_back_addr":"172.21.15.139:6823/2731115256","heartbeat_front_addr":"172.21.15.139:6821/2731115256","state":["exists","up"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6825","nonce":76637888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6827","nonce":76637888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6831","nonce":76637888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6829","nonce":76637888}]},"public_addr":"172.21.15.144:6825/76637888","cluster_addr":"172.21.15.144:6827/76637888","heartbeat_back_addr":"172.21.15.144:6831/76637888","heartbeat_front_addr":"172.21.15.144:6829/76637888","state":["exists","up"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6803","nonce":58455506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6805","nonce":58455506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6809","nonce":58455506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6807","nonce":58455506}]},"public_addr":"172.21.15.165:6803/58455506","cluster_addr":"172.21.15.165:6805/58455506","heartbeat_back_addr":"172.21.15.165:6809/58455506","heartbeat_front_addr":"172.21.15.165:6807/58455506","state":["exists","up"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6825","nonce":2712399685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6827","nonce":2712399685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6831","nonce":2712399685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6829","nonce":2712399685}]},"public_addr":"172.21.15.59:6825/2712399685","cluster_addr":"172.21.15.59:6827/2712399685","heartbeat_back_addr":"172.21.15.59:6831/2712399685","heartbeat_front_addr":"172.21.15.59:6829/2712399685","state":["exists","up"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6801","nonce":2946170755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6803","nonce":2946170755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6807","nonce":2946170755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6805","nonce":2946170755}]},"public_addr":"172.21.15.139:6801/2946170755","cluster_addr":"172.21.15.139:6803/2946170755","heartbeat_back_addr":"172.21.15.139:6807/2946170755","heartbeat_front_addr":"172.21.15.139:6805/2946170755","state":["exists","up"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6801","nonce":3812816248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6803","nonce":3812816248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6807","nonce":3812816248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6805","nonce":3812816248}]},"public_addr":"172.21.15.144:6801/3812816248","cluster_addr":"172.21.15.144:6803/3812816248","heartbeat_back_addr":"172.21.15.144:6807/3812816248","heartbeat_front_addr":"172.21.15.144:6805/3812816248","state":["exists","up"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6819","nonce":1352862134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6821","nonce":1352862134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6825","nonce":1352862134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6823","nonce":1352862134}]},"public_addr":"172.21.15.165:6819/1352862134","cluster_addr":"172.21.15.165:6821/1352862134","heartbeat_back_addr":"172.21.15.165:6825/1352862134","heartbeat_front_addr":"172.21.15.165:6823/1352862134","state":["exists","up"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6817","nonce":1451673997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6819","nonce":1451673997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6823","nonce":1451673997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6821","nonce":1451673997}]},"public_addr":"172.21.15.59:6817/1451673997","cluster_addr":"172.21.15.59:6819/1451673997","heartbeat_back_addr":"172.21.15.59:6823/1451673997","heartbeat_front_addr":"172.21.15.59:6821/1451673997","state":["exists","up"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6809","nonce":174106571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6811","nonce":174106571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6815","nonce":174106571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6813","nonce":174106571}]},"public_addr":"172.21.15.139:6809/174106571","cluster_addr":"172.21.15.139:6811/174106571","heartbeat_back_addr":"172.21.15.139:6815/174106571","heartbeat_front_addr":"172.21.15.139:6813/174106571","state":["exists","up"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6809","nonce":66779937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6812","nonce":66779937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6818","nonce":66779937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6815","nonce":66779937}]},"public_addr":"172.21.15.144:6809/66779937","cluster_addr":"172.21.15.144:6812/66779937","heartbeat_back_addr":"172.21.15.144:6818/66779937","heartbeat_front_addr":"172.21.15.144:6815/66779937","state":["exists","up"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6811","nonce":49204916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6813","nonce":49204916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6817","nonce":49204916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6815","nonce":49204916}]},"public_addr":"172.21.15.165:6811/49204916","cluster_addr":"172.21.15.165:6813/49204916","heartbeat_back_addr":"172.21.15.165:6817/49204916","heartbeat_front_addr":"172.21.15.165:6815/49204916","state":["exists","up"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6809","nonce":3097346146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6811","nonce":3097346146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6815","nonce":3097346146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6813","nonce":3097346146}]},"public_addr":"172.21.15.59:6809/3097346146","cluster_addr":"172.21.15.59:6811/3097346146","heartbeat_back_addr":"172.21.15.59:6815/3097346146","heartbeat_front_addr":"172.21.15.59:6813/3097346146","state":["exists","up"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6825","nonce":2903817823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6827","nonce":2903817823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6831","nonce":2903817823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6829","nonce":2903817823}]},"public_addr":"172.21.15.139:6825/2903817823","cluster_addr":"172.21.15.139:6827/2903817823","heartbeat_back_addr":"172.21.15.139:6831/2903817823","heartbeat_front_addr":"172.21.15.139:6829/2903817823","state":["exists","up"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6811","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6814","nonce":229023981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6819","nonce":229023981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6823","nonce":229023981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6821","nonce":229023981}]},"public_addr":"172.21.15.144:6814/229023981","cluster_addr":"172.21.15.144:6819/229023981","heartbeat_back_addr":"172.21.15.144:6823/229023981","heartbeat_front_addr":"172.21.15.144:6821/229023981","state":["exists","up"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6827","nonce":2758676961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6829","nonce":2758676961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6833","nonce":2758676961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6831","nonce":2758676961}]},"public_addr":"172.21.15.165:6827/2758676961","cluster_addr":"172.21.15.165:6829/2758676961","heartbeat_back_addr":"172.21.15.165:6833/2758676961","heartbeat_front_addr":"172.21.15.165:6831/2758676961","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.223183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.481924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.389820+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.279164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.793764+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.571561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.265206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.684523+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.452789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.332716+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.258982+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.166555+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.256846+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.572294+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.852615+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:41:21.867981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:41:30.692 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:41:30.692 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:41:30.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:41:30.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:41:30.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:41:30.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:41:30.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T04:41:30.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T04:41:30.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T04:41:30.696 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T04:41:30.696 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T04:41:30.697 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T04:41:30.697 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T04:41:30.698 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T04:41:30.698 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T04:41:30.698 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T04:41:31.074 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:31.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T04:41:31.137 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:31.138 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:41:31.160 INFO:teuthology.orchestra.run.smithi059.stdout:90194313220 2024-09-16T04:41:31.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:41:31.171 INFO:teuthology.orchestra.run.smithi059.stdout:98784247814 2024-09-16T04:41:31.172 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:41:31.175 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:31.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T04:41:31.212 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:31.212 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:41:31.241 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:31.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T04:41:31.253 INFO:teuthology.orchestra.run.smithi059.stdout:90194313220 2024-09-16T04:41:31.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:41:31.257 INFO:teuthology.orchestra.run.smithi059.stdout:98784247814 2024-09-16T04:41:31.257 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T04:41:31.258 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:31.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:41:31.277 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:31.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:41:31.278 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:31.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:41:31.287 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:31.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:41:31.288 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:31.289 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:41:31.300 INFO:teuthology.orchestra.run.smithi059.stdout:98784247814 2024-09-16T04:41:31.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:41:31.307 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:31.308 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:41:31.955 INFO:teuthology.orchestra.run.smithi059.stdout:98784247813 2024-09-16T04:41:31.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.9 2024-09-16T04:41:32.002 INFO:teuthology.orchestra.run.smithi059.stdout:94489280515 2024-09-16T04:41:32.055 INFO:teuthology.orchestra.run.smithi059.stdout:98784247813 2024-09-16T04:41:32.066 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.15 2024-09-16T04:41:32.092 INFO:teuthology.orchestra.run.smithi059.stdout:90194313219 2024-09-16T04:41:32.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.1 2024-09-16T04:41:32.108 INFO:teuthology.orchestra.run.smithi059.stdout:94489280515 2024-09-16T04:41:32.142 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.0 2024-09-16T04:41:32.160 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.6 2024-09-16T04:41:32.241 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:32.256 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:32.261 INFO:teuthology.orchestra.run.smithi059.stdout:94489280515 2024-09-16T04:41:32.266 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:32.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.14 2024-09-16T04:41:32.300 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.12 2024-09-16T04:41:32.314 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.2 2024-09-16T04:41:32.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.3 2024-09-16T04:41:32.326 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:32.343 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:32.344 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.7 2024-09-16T04:41:32.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.4 2024-09-16T04:41:32.372 INFO:teuthology.orchestra.run.smithi059.stdout:94489280515 2024-09-16T04:41:32.376 INFO:teuthology.orchestra.run.smithi059.stdout:98784247813 2024-09-16T04:41:32.383 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:32.402 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.8 2024-09-16T04:41:32.403 INFO:teuthology.orchestra.run.smithi059.stdout:90194313219 2024-09-16T04:41:32.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.13 2024-09-16T04:41:32.410 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.10 2024-09-16T04:41:32.413 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:32.423 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.11 2024-09-16T04:41:32.432 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.5 2024-09-16T04:41:32.995 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:41:33.067 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T04:41:33.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:41:33.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:41:33.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:41:33.291 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T04:41:33.301 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T04:41:33.315 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:41:33.322 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:41:33.331 INFO:teuthology.orchestra.run.smithi059.stdout:98784247813 2024-09-16T04:41:33.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:41:33.357 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.9 2024-09-16T04:41:33.363 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:41:33.403 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:41:33.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T04:41:33.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:41:33.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:41:33.433 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:41:33.485 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:33.522 INFO:teuthology.orchestra.run.smithi059.stdout:98784247814 2024-09-16T04:41:33.541 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.15 2024-09-16T04:41:33.542 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:33.585 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.1 2024-09-16T04:41:33.585 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:33.802 INFO:teuthology.orchestra.run.smithi059.stdout:90194313221 2024-09-16T04:41:33.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313221 for osd.0 2024-09-16T04:41:33.867 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:33.972 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:34.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.6 2024-09-16T04:41:34.029 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.043 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:34.049 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:34.050 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:34.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.12 2024-09-16T04:41:34.088 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.14 2024-09-16T04:41:34.101 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.2 2024-09-16T04:41:34.103 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.117 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:34.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.3 2024-09-16T04:41:34.142 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.155 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:34.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.10 2024-09-16T04:41:34.182 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.207 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:34.231 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.4 2024-09-16T04:41:34.231 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.232 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:34.241 INFO:teuthology.orchestra.run.smithi059.stdout:90194313221 2024-09-16T04:41:34.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.7 2024-09-16T04:41:34.251 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.262 INFO:teuthology.orchestra.run.smithi059.stdout:94489280517 2024-09-16T04:41:34.264 INFO:teuthology.orchestra.run.smithi059.stdout:94489280516 2024-09-16T04:41:34.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313221 for osd.11 2024-09-16T04:41:34.266 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.282 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.5 2024-09-16T04:41:34.282 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.283 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.8 2024-09-16T04:41:34.283 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.292 INFO:teuthology.orchestra.run.smithi059.stdout:98784247814 2024-09-16T04:41:34.309 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.13 2024-09-16T04:41:34.310 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.358 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:41:34.642 INFO:teuthology.orchestra.run.smithi059.stdout:98784247814 2024-09-16T04:41:34.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.9 2024-09-16T04:41:34.661 DEBUG:teuthology.parallel:result is None 2024-09-16T04:41:34.661 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:41:34.661 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:34.949 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:34.950 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:41:34.966 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T04:41:32.987149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436824,"kb_used_data":4348,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512616,"statfs":{"total":1546188226560,"available":1545740918784,"internally_reserved":0,"allocated":4452352,"data_stored":2391398,"data_compressed":292054,"data_compressed_allocated":421888,"data_compressed_original":1245184,"omap_allocated":25522,"internal_metadata":442473550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459291,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":91,"num_write_kb":982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.572701"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.107799+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:30.107799+0000","last_peered":"2024-09-16T04:41:30.107799+0000","last_clean":"2024-09-16T04:41:30.107799+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:30.107799+0000","last_undegraded":"2024-09-16T04:41:30.107799+0000","last_fullsized":"2024-09-16T04:41:30.107799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.071835+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:30.071835+0000","last_peered":"2024-09-16T04:41:30.071835+0000","last_clean":"2024-09-16T04:41:30.071835+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:30.071835+0000","last_undegraded":"2024-09-16T04:41:30.071835+0000","last_fullsized":"2024-09-16T04:41:30.071835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.109464+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:30.109464+0000","last_peered":"2024-09-16T04:41:30.109464+0000","last_clean":"2024-09-16T04:41:30.109464+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:30.109464+0000","last_undegraded":"2024-09-16T04:41:30.109464+0000","last_fullsized":"2024-09-16T04:41:30.109464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.769679+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:29.769679+0000","last_peered":"2024-09-16T04:41:29.769679+0000","last_clean":"2024-09-16T04:41:29.769679+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:29.769679+0000","last_undegraded":"2024-09-16T04:41:29.769679+0000","last_fullsized":"2024-09-16T04:41:29.769679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.619423+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:29.619423+0000","last_peered":"2024-09-16T04:41:29.619423+0000","last_clean":"2024-09-16T04:41:29.619423+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:29.619423+0000","last_undegraded":"2024-09-16T04:41:29.619423+0000","last_fullsized":"2024-09-16T04:41:29.619423+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209135+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:41:30.209135+0000","last_peered":"2024-09-16T04:41:30.209135+0000","last_clean":"2024-09-16T04:41:30.209135+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:41:30.209135+0000","last_undegraded":"2024-09-16T04:41:30.209135+0000","last_fullsized":"2024-09-16T04:41:30.209135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.518316+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:29.518316+0000","last_peered":"2024-09-16T04:41:29.518316+0000","last_clean":"2024-09-16T04:41:29.518316+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:29.518316+0000","last_undegraded":"2024-09-16T04:41:29.518316+0000","last_fullsized":"2024-09-16T04:41:29.518316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209227+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:30.209227+0000","last_peered":"2024-09-16T04:41:30.209227+0000","last_clean":"2024-09-16T04:41:30.209227+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:30.209227+0000","last_undegraded":"2024-09-16T04:41:30.209227+0000","last_fullsized":"2024-09-16T04:41:30.209227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'58","reported_seq":96,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:31.082932+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:41:31.082932+0000","last_peered":"2024-09-16T04:41:31.082932+0000","last_clean":"2024-09-16T04:41:31.082932+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:41:31.082932+0000","last_undegraded":"2024-09-16T04:41:31.082932+0000","last_fullsized":"2024-09-16T04:41:31.082932+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":795680,"data_compressed":37602,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":303104,"data_stored":390562,"data_compressed":29629,"data_compressed_allocated":40960,"data_compressed_original":311296,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":299008,"data_stored":325026,"data_compressed":26539,"data_compressed_allocated":36864,"data_compressed_original":245760,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":18802,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":18800,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:34.968 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:35.268 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:41:35.286 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T04:41:35.065450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436844,"kb_used_data":4368,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512596,"statfs":{"total":1546188226560,"available":1545740898304,"internally_reserved":0,"allocated":4472832,"data_stored":2591586,"data_compressed":301151,"data_compressed_allocated":434176,"data_compressed_original":1437696,"omap_allocated":25522,"internal_metadata":442473550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459291,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":91,"num_write_kb":982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.651002"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.107799+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:30.107799+0000","last_peered":"2024-09-16T04:41:30.107799+0000","last_clean":"2024-09-16T04:41:30.107799+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:30.107799+0000","last_undegraded":"2024-09-16T04:41:30.107799+0000","last_fullsized":"2024-09-16T04:41:30.107799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.071835+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:30.071835+0000","last_peered":"2024-09-16T04:41:30.071835+0000","last_clean":"2024-09-16T04:41:30.071835+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:30.071835+0000","last_undegraded":"2024-09-16T04:41:30.071835+0000","last_fullsized":"2024-09-16T04:41:30.071835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.109464+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:30.109464+0000","last_peered":"2024-09-16T04:41:30.109464+0000","last_clean":"2024-09-16T04:41:30.109464+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:30.109464+0000","last_undegraded":"2024-09-16T04:41:30.109464+0000","last_fullsized":"2024-09-16T04:41:30.109464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.769679+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:29.769679+0000","last_peered":"2024-09-16T04:41:29.769679+0000","last_clean":"2024-09-16T04:41:29.769679+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:29.769679+0000","last_undegraded":"2024-09-16T04:41:29.769679+0000","last_fullsized":"2024-09-16T04:41:29.769679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.619423+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:29.619423+0000","last_peered":"2024-09-16T04:41:29.619423+0000","last_clean":"2024-09-16T04:41:29.619423+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:29.619423+0000","last_undegraded":"2024-09-16T04:41:29.619423+0000","last_fullsized":"2024-09-16T04:41:29.619423+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209135+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:41:30.209135+0000","last_peered":"2024-09-16T04:41:30.209135+0000","last_clean":"2024-09-16T04:41:30.209135+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:41:30.209135+0000","last_undegraded":"2024-09-16T04:41:30.209135+0000","last_fullsized":"2024-09-16T04:41:30.209135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.518316+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:29.518316+0000","last_peered":"2024-09-16T04:41:29.518316+0000","last_clean":"2024-09-16T04:41:29.518316+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:29.518316+0000","last_undegraded":"2024-09-16T04:41:29.518316+0000","last_fullsized":"2024-09-16T04:41:29.518316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209227+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:30.209227+0000","last_peered":"2024-09-16T04:41:30.209227+0000","last_clean":"2024-09-16T04:41:30.209227+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:30.209227+0000","last_undegraded":"2024-09-16T04:41:30.209227+0000","last_fullsized":"2024-09-16T04:41:30.209227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'58","reported_seq":96,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:31.082932+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:41:31.082932+0000","last_peered":"2024-09-16T04:41:31.082932+0000","last_clean":"2024-09-16T04:41:31.082932+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:41:31.082932+0000","last_undegraded":"2024-09-16T04:41:31.082932+0000","last_fullsized":"2024-09-16T04:41:31.082932+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":803356,"data_compressed":40386,"data_compressed_allocated":53248,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":303104,"data_stored":390562,"data_compressed":29629,"data_compressed_allocated":40960,"data_compressed_original":311296,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":525214,"data_compressed":35636,"data_compressed_allocated":49152,"data_compressed_original":438272,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":405516,"data_compressed":21586,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":18800,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:35.287 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:41:35.288 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T04:41:35.288 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T04:41:35.288 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T04:41:35.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:35.609 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T04:41:35.627 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T04:41:35.627 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T04:41:35.638 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T04:41:35.638 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}} 2024-09-16T04:41:35.638 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T04:41:35.639 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T04:41:35.950 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T04:41:35.950 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:41:35.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:36.250 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:36.250 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:41:36.268 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T04:41:35.065450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436844,"kb_used_data":4368,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512596,"statfs":{"total":1546188226560,"available":1545740898304,"internally_reserved":0,"allocated":4472832,"data_stored":2591586,"data_compressed":301151,"data_compressed_allocated":434176,"data_compressed_original":1437696,"omap_allocated":25522,"internal_metadata":442473550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459291,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":91,"num_write_kb":982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.651002"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.107799+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:30.107799+0000","last_peered":"2024-09-16T04:41:30.107799+0000","last_clean":"2024-09-16T04:41:30.107799+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:30.107799+0000","last_undegraded":"2024-09-16T04:41:30.107799+0000","last_fullsized":"2024-09-16T04:41:30.107799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.071835+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:30.071835+0000","last_peered":"2024-09-16T04:41:30.071835+0000","last_clean":"2024-09-16T04:41:30.071835+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:30.071835+0000","last_undegraded":"2024-09-16T04:41:30.071835+0000","last_fullsized":"2024-09-16T04:41:30.071835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.109464+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:30.109464+0000","last_peered":"2024-09-16T04:41:30.109464+0000","last_clean":"2024-09-16T04:41:30.109464+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:30.109464+0000","last_undegraded":"2024-09-16T04:41:30.109464+0000","last_fullsized":"2024-09-16T04:41:30.109464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.769679+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:29.769679+0000","last_peered":"2024-09-16T04:41:29.769679+0000","last_clean":"2024-09-16T04:41:29.769679+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:29.769679+0000","last_undegraded":"2024-09-16T04:41:29.769679+0000","last_fullsized":"2024-09-16T04:41:29.769679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.619423+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:29.619423+0000","last_peered":"2024-09-16T04:41:29.619423+0000","last_clean":"2024-09-16T04:41:29.619423+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:29.619423+0000","last_undegraded":"2024-09-16T04:41:29.619423+0000","last_fullsized":"2024-09-16T04:41:29.619423+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209135+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:41:30.209135+0000","last_peered":"2024-09-16T04:41:30.209135+0000","last_clean":"2024-09-16T04:41:30.209135+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:41:30.209135+0000","last_undegraded":"2024-09-16T04:41:30.209135+0000","last_fullsized":"2024-09-16T04:41:30.209135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.518316+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:29.518316+0000","last_peered":"2024-09-16T04:41:29.518316+0000","last_clean":"2024-09-16T04:41:29.518316+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:29.518316+0000","last_undegraded":"2024-09-16T04:41:29.518316+0000","last_fullsized":"2024-09-16T04:41:29.518316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209227+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:30.209227+0000","last_peered":"2024-09-16T04:41:30.209227+0000","last_clean":"2024-09-16T04:41:30.209227+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:30.209227+0000","last_undegraded":"2024-09-16T04:41:30.209227+0000","last_fullsized":"2024-09-16T04:41:30.209227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'58","reported_seq":96,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:31.082932+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:41:31.082932+0000","last_peered":"2024-09-16T04:41:31.082932+0000","last_clean":"2024-09-16T04:41:31.082932+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:41:31.082932+0000","last_undegraded":"2024-09-16T04:41:31.082932+0000","last_fullsized":"2024-09-16T04:41:31.082932+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":803356,"data_compressed":40386,"data_compressed_allocated":53248,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":303104,"data_stored":390562,"data_compressed":29629,"data_compressed_allocated":40960,"data_compressed_original":311296,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":525214,"data_compressed":35636,"data_compressed_allocated":49152,"data_compressed_original":438272,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":405516,"data_compressed":21586,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":18800,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:36.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:36.569 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:36.569 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:41:36.586 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T04:41:35.065450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":436844,"kb_used_data":4368,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512596,"statfs":{"total":1546188226560,"available":1545740898304,"internally_reserved":0,"allocated":4472832,"data_stored":2591586,"data_compressed":301151,"data_compressed_allocated":434176,"data_compressed_original":1437696,"omap_allocated":25522,"internal_metadata":442473550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459291,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":91,"num_write_kb":982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.651002"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.107799+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:30.107799+0000","last_peered":"2024-09-16T04:41:30.107799+0000","last_clean":"2024-09-16T04:41:30.107799+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:30.107799+0000","last_undegraded":"2024-09-16T04:41:30.107799+0000","last_fullsized":"2024-09-16T04:41:30.107799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.071835+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:30.071835+0000","last_peered":"2024-09-16T04:41:30.071835+0000","last_clean":"2024-09-16T04:41:30.071835+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:30.071835+0000","last_undegraded":"2024-09-16T04:41:30.071835+0000","last_fullsized":"2024-09-16T04:41:30.071835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.109464+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:30.109464+0000","last_peered":"2024-09-16T04:41:30.109464+0000","last_clean":"2024-09-16T04:41:30.109464+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:30.109464+0000","last_undegraded":"2024-09-16T04:41:30.109464+0000","last_fullsized":"2024-09-16T04:41:30.109464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.769679+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:29.769679+0000","last_peered":"2024-09-16T04:41:29.769679+0000","last_clean":"2024-09-16T04:41:29.769679+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:29.769679+0000","last_undegraded":"2024-09-16T04:41:29.769679+0000","last_fullsized":"2024-09-16T04:41:29.769679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.619423+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:29.619423+0000","last_peered":"2024-09-16T04:41:29.619423+0000","last_clean":"2024-09-16T04:41:29.619423+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:29.619423+0000","last_undegraded":"2024-09-16T04:41:29.619423+0000","last_fullsized":"2024-09-16T04:41:29.619423+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209135+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:41:30.209135+0000","last_peered":"2024-09-16T04:41:30.209135+0000","last_clean":"2024-09-16T04:41:30.209135+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:41:30.209135+0000","last_undegraded":"2024-09-16T04:41:30.209135+0000","last_fullsized":"2024-09-16T04:41:30.209135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:29.518316+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:29.518316+0000","last_peered":"2024-09-16T04:41:29.518316+0000","last_clean":"2024-09-16T04:41:29.518316+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:29.518316+0000","last_undegraded":"2024-09-16T04:41:29.518316+0000","last_fullsized":"2024-09-16T04:41:29.518316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:30.209227+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:30.209227+0000","last_peered":"2024-09-16T04:41:30.209227+0000","last_clean":"2024-09-16T04:41:30.209227+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:30.209227+0000","last_undegraded":"2024-09-16T04:41:30.209227+0000","last_fullsized":"2024-09-16T04:41:30.209227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'58","reported_seq":96,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:41:31.082932+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:41:31.082932+0000","last_peered":"2024-09-16T04:41:31.082932+0000","last_clean":"2024-09-16T04:41:31.082932+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:41:31.082932+0000","last_undegraded":"2024-09-16T04:41:31.082932+0000","last_fullsized":"2024-09-16T04:41:31.082932+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":90,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":803356,"data_compressed":40386,"data_compressed_allocated":53248,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":303104,"data_stored":390562,"data_compressed":29629,"data_compressed_allocated":40960,"data_compressed_original":311296,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":525214,"data_compressed":35636,"data_compressed_allocated":49152,"data_compressed_original":438272,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":119717,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":119698,"data_compressed":16849,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":405516,"data_compressed":21586,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":18800,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:36.588 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:41:36.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T04:41:36.885 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 29 2024-09-16T04:41:36.885 INFO:teuthology.orchestra.run.smithi059.stdout:fsid bdbec34e-c8aa-4776-9d32-ea336a99655f 2024-09-16T04:41:36.885 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-09-16T04:41:10.374765+0000 2024-09-16T04:41:36.885 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-09-16T04:41:35.062208+0000 2024-09-16T04:41:36.885 INFO:teuthology.orchestra.run.smithi059.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T04:41:36.885 INFO:teuthology.orchestra.run.smithi059.stdout:crush_version 5 2024-09-16T04:41:36.885 INFO:teuthology.orchestra.run.smithi059.stdout:full_ratio 0.95 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:backfillfull_ratio 0.9 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:nearfull_ratio 0.85 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:require_min_compat_client luminous 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:min_compat_client jewel 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:require_osd_release squid 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:stretch_mode_enabled false 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.02 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:max_osd 16 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6800/1228226348,v1:172.21.15.59:6801/1228226348] [v2:172.21.15.59:6802/1228226348,v1:172.21.15.59:6803/1228226348] exists,up b9529dc0-e1eb-4b2f-8297-45254ddbffa8 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6816/2731115256,v1:172.21.15.139:6817/2731115256] [v2:172.21.15.139:6818/2731115256,v1:172.21.15.139:6819/2731115256] exists,up dceca258-f5a1-4e8d-be6e-9d38bc5f14e1 2024-09-16T04:41:36.886 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 up in weight 1 up_from 22 up_thru 22 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6824/76637888,v1:172.21.15.144:6825/76637888] [v2:172.21.15.144:6826/76637888,v1:172.21.15.144:6827/76637888] exists,up 6bfc934a-0ca2-4b1f-9e09-14c18a7d5116 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6802/58455506,v1:172.21.15.165:6803/58455506] [v2:172.21.15.165:6804/58455506,v1:172.21.15.165:6805/58455506] exists,up 0b949ef9-82e3-483f-8be6-65e04d169c4e 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6824/2712399685,v1:172.21.15.59:6825/2712399685] [v2:172.21.15.59:6826/2712399685,v1:172.21.15.59:6827/2712399685] exists,up d35e5791-7bb4-493d-99ec-3ff942117371 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/2946170755,v1:172.21.15.139:6801/2946170755] [v2:172.21.15.139:6802/2946170755,v1:172.21.15.139:6803/2946170755] exists,up 159725f2-3713-4f0e-9e44-55b678bf09fa 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6800/3812816248,v1:172.21.15.144:6801/3812816248] [v2:172.21.15.144:6802/3812816248,v1:172.21.15.144:6803/3812816248] exists,up e642cfb3-cf94-4124-93db-9e2bdf1b7c5f 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6818/1352862134,v1:172.21.15.165:6819/1352862134] [v2:172.21.15.165:6820/1352862134,v1:172.21.15.165:6821/1352862134] exists,up 8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6816/1451673997,v1:172.21.15.59:6817/1451673997] [v2:172.21.15.59:6818/1451673997,v1:172.21.15.59:6819/1451673997] exists,up 04681c69-387b-4dd8-9eb4-687f34ba9dab 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/174106571,v1:172.21.15.139:6809/174106571] [v2:172.21.15.139:6810/174106571,v1:172.21.15.139:6811/174106571] exists,up 713a3b76-8daf-4e44-9d0b-68369a9da692 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6808/66779937,v1:172.21.15.144:6809/66779937] [v2:172.21.15.144:6810/66779937,v1:172.21.15.144:6812/66779937] exists,up 81450492-3178-4643-8066-847fcdd266b4 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6810/49204916,v1:172.21.15.165:6811/49204916] [v2:172.21.15.165:6812/49204916,v1:172.21.15.165:6813/49204916] exists,up 9938b272-43e2-4b48-91ca-42cbe5f26664 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6808/3097346146,v1:172.21.15.59:6809/3097346146] [v2:172.21.15.59:6810/3097346146,v1:172.21.15.59:6811/3097346146] exists,up cd20410e-c17f-43d7-8512-9efd8193a251 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6824/2903817823,v1:172.21.15.139:6825/2903817823] [v2:172.21.15.139:6826/2903817823,v1:172.21.15.139:6827/2903817823] exists,up 953622d2-9e72-451d-8863-d8c55139c65b 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6811/229023981,v1:172.21.15.144:6814/229023981] [v2:172.21.15.144:6816/229023981,v1:172.21.15.144:6819/229023981] exists,up e6388924-0444-47c0-b61c-7ca66661237f 2024-09-16T04:41:36.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6826/2758676961,v1:172.21.15.165:6827/2758676961] [v2:172.21.15.165:6828/2758676961,v1:172.21.15.165:6829/2758676961] exists,up 8335941d-9563-4527-90f6-3b7f9129f530 2024-09-16T04:41:36.903 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6800/1228226348,v1:172.21.15.59:6801/1228226348] [v2:172.21.15.59:6802/1228226348,v1:172.21.15.59:6803/1228226348] exists,up b9529dc0-e1eb-4b2f-8297-45254ddbffa8', 'osd.1 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6816/2731115256,v1:172.21.15.139:6817/2731115256] [v2:172.21.15.139:6818/2731115256,v1:172.21.15.139:6819/2731115256] exists,up dceca258-f5a1-4e8d-be6e-9d38bc5f14e1', 'osd.2 up in weight 1 up_from 22 up_thru 22 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6824/76637888,v1:172.21.15.144:6825/76637888] [v2:172.21.15.144:6826/76637888,v1:172.21.15.144:6827/76637888] exists,up 6bfc934a-0ca2-4b1f-9e09-14c18a7d5116', 'osd.3 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6802/58455506,v1:172.21.15.165:6803/58455506] [v2:172.21.15.165:6804/58455506,v1:172.21.15.165:6805/58455506] exists,up 0b949ef9-82e3-483f-8be6-65e04d169c4e', 'osd.4 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6824/2712399685,v1:172.21.15.59:6825/2712399685] [v2:172.21.15.59:6826/2712399685,v1:172.21.15.59:6827/2712399685] exists,up d35e5791-7bb4-493d-99ec-3ff942117371', 'osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/2946170755,v1:172.21.15.139:6801/2946170755] [v2:172.21.15.139:6802/2946170755,v1:172.21.15.139:6803/2946170755] exists,up 159725f2-3713-4f0e-9e44-55b678bf09fa', 'osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6800/3812816248,v1:172.21.15.144:6801/3812816248] [v2:172.21.15.144:6802/3812816248,v1:172.21.15.144:6803/3812816248] exists,up e642cfb3-cf94-4124-93db-9e2bdf1b7c5f', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6818/1352862134,v1:172.21.15.165:6819/1352862134] [v2:172.21.15.165:6820/1352862134,v1:172.21.15.165:6821/1352862134] exists,up 8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6816/1451673997,v1:172.21.15.59:6817/1451673997] [v2:172.21.15.59:6818/1451673997,v1:172.21.15.59:6819/1451673997] exists,up 04681c69-387b-4dd8-9eb4-687f34ba9dab', 'osd.9 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/174106571,v1:172.21.15.139:6809/174106571] [v2:172.21.15.139:6810/174106571,v1:172.21.15.139:6811/174106571] exists,up 713a3b76-8daf-4e44-9d0b-68369a9da692', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6808/66779937,v1:172.21.15.144:6809/66779937] [v2:172.21.15.144:6810/66779937,v1:172.21.15.144:6812/66779937] exists,up 81450492-3178-4643-8066-847fcdd266b4', 'osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6810/49204916,v1:172.21.15.165:6811/49204916] [v2:172.21.15.165:6812/49204916,v1:172.21.15.165:6813/49204916] exists,up 9938b272-43e2-4b48-91ca-42cbe5f26664', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6808/3097346146,v1:172.21.15.59:6809/3097346146] [v2:172.21.15.59:6810/3097346146,v1:172.21.15.59:6811/3097346146] exists,up cd20410e-c17f-43d7-8512-9efd8193a251', 'osd.13 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6824/2903817823,v1:172.21.15.139:6825/2903817823] [v2:172.21.15.139:6826/2903817823,v1:172.21.15.139:6827/2903817823] exists,up 953622d2-9e72-451d-8863-d8c55139c65b', 'osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6811/229023981,v1:172.21.15.144:6814/229023981] [v2:172.21.15.144:6816/229023981,v1:172.21.15.144:6819/229023981] exists,up e6388924-0444-47c0-b61c-7ca66661237f', 'osd.15 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6826/2758676961,v1:172.21.15.165:6827/2758676961] [v2:172.21.15.165:6828/2758676961,v1:172.21.15.165:6829/2758676961] exists,up 8335941d-9563-4527-90f6-3b7f9129f530'] 2024-09-16T04:41:36.904 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "mon.a", 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster": "ceph", 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout: "admin_socket_mode": "", 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_client_required": "cephx, none", 2024-09-16T04:41:37.025 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_cluster_required": "cephx", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_debug": "true", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_service_required": "cephx", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_supported": "cephx", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio": "true", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_reap_max": "16", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_async_discard": "false", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_block_size": "4096", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio": "false", 2024-09-16T04:41:37.026 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_enable_discard": "false", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_flock_retry": "3", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_inject_crash": "0", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring": "false", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring_hipri": "false", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T04:41:37.027 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_type": "", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_allocator": "hybrid", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_buffered_io": "true", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T04:41:37.028 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_replay_recovery": "false", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_sync_write": "false", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T04:41:37.029 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_allocator": "hybrid", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_create": "true", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_create": "false", 2024-09-16T04:41:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_path": "", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_size": "0", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_path": "", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_size": "107374182400", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_create": "false", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_path": "", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs": "true", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T04:41:37.031 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_autotune": "true", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size": "0", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T04:41:37.032 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_type": "2q", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_clone_cow": "true", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_mode": "none", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T04:41:37.033 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_freelist": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T04:41:37.034 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T04:41:37.035 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fail_eio": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T04:41:37.036 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size": "0", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T04:41:37.037 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T04:41:37.038 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_mem": "512", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T04:41:37.039 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T04:41:37.040 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T04:41:37.040 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_tracing": "false", 2024-09-16T04:41:37.040 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T04:41:37.040 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T04:41:37.040 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T04:41:37.040 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T04:41:37.041 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T04:41:37.042 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T04:41:37.042 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T04:41:37.042 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T04:41:37.042 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T04:41:37.042 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T04:41:37.042 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T04:41:37.042 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T04:41:37.043 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_cluster_require_version": "2", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_require_signatures": "false", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_require_version": "2", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_service_require_signatures": "false", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_service_require_version": "2", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_sign_messages": "true", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "chdir": "", 2024-09-16T04:41:37.044 INFO:teuthology.orchestra.run.smithi059.stdout: "client_acl_type": "", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_asio_thread_count": "2", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_cache_mid": "0.750000", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_cache_size": "16384", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_caps_release_delay": "5", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_check_pool_perm": "true", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_force_sync_read": "false", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_getattr_caps": "false", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_inject_features": "", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_die_on_failed_remount": "false", 2024-09-16T04:41:37.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_dirsize_rbytes": "true", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_force_lazyio": "false", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_fs": "", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_inject_release_failure": "false", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_max_inline_size": "4096", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mds_namespace": "", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_metadata": "", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_gid": "-1", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_timeout": "300", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_uid": "-1", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mountpoint": "/", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_notify_timeout": "10", 2024-09-16T04:41:37.046 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc": "true", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_objects": "1000", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_size": "209715200", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_permissions": "true", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_quota": "true", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_quota_df": "true", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_max_bytes": "0", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_max_periods": "4", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_min": "131072", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_reconnect_stale": "false", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_shutdown_timeout": "30", 2024-09-16T04:41:37.047 INFO:teuthology.orchestra.run.smithi059.stdout: "client_snapdir": ".snap", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "client_tick_interval": "1", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "client_trace": "", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "client_use_faked_inos": "false", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "client_use_random_mds": "false", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog": "false", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog_port": "12201", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_monitors": "default=true", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog": "false", 2024-09-16T04:41:37.048 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog_level": "info", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "-", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_network": "", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_network_interface": "", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_isal": "false", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_level": "5", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zstd_level": "1", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T04:41:37.049 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location": "", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location_hook": "", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location_hook_timeout": "10", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "daemonize": "false", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "daos_pool": "tank", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_alienstore": "0/5", 2024-09-16T04:41:37.050 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asok": "1/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asok_assert_abort": "false", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_auth": "20/20", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bdev": "1/3", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bluefs": "1/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bluestore": "1/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_buffer": "0/1", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cephsqlite": "0/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_client": "0/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_compressor": "1/5", 2024-09-16T04:41:37.051 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_context": "0/1", 2024-09-16T04:41:37.052 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T04:41:37.052 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crush": "1/1", 2024-09-16T04:41:37.052 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crypto": "1/5", 2024-09-16T04:41:37.052 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cyanstore": "0/5", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_dpdk": "1/5", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_eventtrace": "1/5", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_filer": "0/1", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_filestore": "1/3", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_finisher": "1/1", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_fuse": "1/5", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T04:41:37.053 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T04:41:37.054 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_javaclient": "1/5", 2024-09-16T04:41:37.054 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_journal": "1/3", 2024-09-16T04:41:37.054 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_journaler": "0/5", 2024-09-16T04:41:37.054 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_kstore": "1/5", 2024-09-16T04:41:37.055 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_lockdep": "0/1", 2024-09-16T04:41:37.055 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mclock": "1/5", 2024-09-16T04:41:37.055 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds": "1/5", 2024-09-16T04:41:37.055 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_balancer": "1/5", 2024-09-16T04:41:37.055 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_locker": "1/5", 2024-09-16T04:41:37.056 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_log": "1/5", 2024-09-16T04:41:37.056 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T04:41:37.057 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_migrator": "1/5", 2024-09-16T04:41:37.057 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T04:41:37.057 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_memstore": "1/5", 2024-09-16T04:41:37.058 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mgr": "2/5", 2024-09-16T04:41:37.058 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mgrc": "1/5", 2024-09-16T04:41:37.058 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mon": "20/20", 2024-09-16T04:41:37.059 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_monc": "0/10", 2024-09-16T04:41:37.059 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_ms": "1/1", 2024-09-16T04:41:37.059 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_none": "0/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objclass": "0/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objectcacher": "0/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objecter": "0/1", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_optracker": "0/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_osd": "1/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_paxos": "20/20", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_perfcounter": "1/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_prioritycache": "1/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rados": "0/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd": "0/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T04:41:37.060 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_replay": "0/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_refs": "0/0", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_reserver": "1/1", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw": "1/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_access": "1/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_flight": "1/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_sync": "1/5", 2024-09-16T04:41:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rocksdb": "4/5", 2024-09-16T04:41:37.062 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore": "0/5", 2024-09-16T04:41:37.062 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_backref": "0/5", 2024-09-16T04:41:37.062 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_cache": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_device": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_epm": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_journal": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_lba": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_odata": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_omap": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_onode": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_t": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_tm": "0/5", 2024-09-16T04:41:37.063 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_striper": "0/1", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_test": "0/5", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_throttle": "1/1", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_timer": "0/1", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_tp": "0/5", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_trace": "1/5", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "device_failure_prediction_mode": "none", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_graylog": "false", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_journald": "false", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_stderr": "true", 2024-09-16T04:41:37.064 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_syslog": "false", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "event_tracing": "false", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_http_port": "9926", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_prio_limit": "5", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_sort_metrics": "true", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_stats_period": "5", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "fake_statfs_for_testing": "0", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "fatal_signal_handlers": "true", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "filer_max_purge_ops": "10", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "filer_max_truncate_ops": "128", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T04:41:37.065 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_blackhole": "false", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_btrfs_snap": "true", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_caller_concurrency": "10", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_omap_check": "false", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_verify_split": "false", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_dump_file": "", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T04:41:37.066 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fadvise": "true", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fail_eio": "true", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fd_cache_size": "128", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fiemap": "false", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_inject_stall": "0", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_parallel": "false", 2024-09-16T04:41:37.067 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_trailing": "false", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_writeahead": "false", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_kill_at": "0", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T04:41:37.068 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_merge_threshold": "-10", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_odsync_write": "false", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_backend_path": "", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T04:41:37.069 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_threads": "2", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_punch_hole": "false", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_ops": "50", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_seek_data_hole": "false", 2024-09-16T04:41:37.070 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_sloppy_crc": "false", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_splice": "false", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_split_multiple": "2", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_split_rand_factor": "20", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_update_to": "1000", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T04:41:37.071 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_xfs_extsize": "false", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_zfs_snap": "false", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "fio_dir": "/tmp/fio", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "fsid": "bdbec34e-c8aa-4776-9d32-ea336a99655f", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_allow_other": "true", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_big_writes": "true", 2024-09-16T04:41:37.072 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_debug": "false", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_default_permissions": "false", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_disable_pagecache": "false", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_max_write": "0", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_multithreaded": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_require_active_mds": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_set_user_groups": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_move": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_read": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_write": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T04:41:37.073 INFO:teuthology.orchestra.run.smithi059.stdout: "gss_target_name": "ceph", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_file": "", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_inject_failure": "0", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_interval": "5", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "host": "smithi059", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T04:41:37.074 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "inject_early_sigterm": "false", 2024-09-16T04:41:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: "jaeger_agent_port": "6799", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "jaeger_tracing_enable": "false", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_aio": "true", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_align_min_size": "65536", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_block_align": "true", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_block_size": "4096", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_dio": "true", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_discard": "false", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_force_aio": "false", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_ignore_corruption": "false", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_max_write_entries": "100", 2024-09-16T04:41:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_replay_from": "0", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_write_header_frequency": "0", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_zero_on_create": "false", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_prefetch_periods": "10", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_prezero_periods": "5", 2024-09-16T04:41:37.077 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_write_head_interval": "15", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "keyfile": "", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_backend": "rocksdb", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_max_bytes": "67108864", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_max_ops": "512", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_onode_map_size": "1024", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T04:41:37.078 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_sync_transaction": "false", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "librados_thread_count": "2", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "lockdep": "false", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "lockdep_force_backtrace": "false", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_coarse_timestamps": "true", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_flush_on_exit": "false", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_graylog_port": "12201", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_max_new": "1000", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_max_recent": "10000", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_stderr_prefix": "", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T04:41:37.079 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_file": "true", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_graylog": "false", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_journald": "false", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_stderr": "false", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_syslog": "false", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_action_on_write_error": "1", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_alternate_name_max": "8192", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_asio_thread_count": "2", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_export_pin": "true", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T04:41:37.080 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_interval": "10", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_max": "-1", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_max_until": "-1", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_merge_size": "50", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_mode": "0", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T04:41:37.081 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_bits": "3", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_size": "10000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_mid": "0.700000", 2024-09-16T04:41:37.082 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_interval": "1", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T04:41:37.083 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_auth_pins": "false", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_frag": "false", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_scatterstat": "false", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_subtrees": "false", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_default_dir_hash": "2", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_defer_session_stale": "true", 2024-09-16T04:41:37.084 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_max_entries": "0", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_prefetch": "true", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_early_reply": "true", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_enable_op_tracker": "true", 2024-09-16T04:41:37.085 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_enforce_unique_name": "true", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T04:41:37.086 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_health_dummy": "false", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_join_fs": "", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_journal_format": "1", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_create_at": "0", 2024-09-16T04:41:37.087 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_export_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_import_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_link_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_openc_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_rename_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T04:41:37.088 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_max_events": "-1", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_max_segments": "128", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_pause": "false", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_segment_size": "0", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_threshold": "128", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_completed_requests": "100000", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_export_size": "20971520", 2024-09-16T04:41:37.089 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_file_recover": "32", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_files": "64", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_ops": "8192", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_metrics_update_interval": "2", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_min_caps_per_client": "100", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_numa_node": "-1", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T04:41:37.090 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_duration": "600", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_size": "20", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_log_threshold": "5", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_ping_grace": "15", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_ping_interval": "5", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_caps": "30000", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T04:41:37.091 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_replay_interval": "1.000000", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_root_ino_gid": "0", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_root_ino_uid": "0", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T04:41:37.092 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_shutdown_check": "0", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_skip_ino": "0", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_min_uid": "0", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_rstat": "false", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T04:41:37.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_symlink_recovery": "true", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_thrash_exports": "0", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_thrash_fragments": "0", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_tick_interval": "5.000000", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_valgrind_exit": "false", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_verify_backtrace": "1", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_verify_scatter": "false", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_wipe_sessions": "false", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "mempool_debug": "false", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T04:41:37.094 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_page_set": "false", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_page_size": "65536", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_bytes": "134217728", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_messages": "512", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_disabled_modules": "", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T04:41:37.095 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mds_messages": "128", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mon_messages": "128", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_op_history_duration": "600", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_op_history_size": "20", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_op_log_threshold": "5", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_osd_messages": "8192", 2024-09-16T04:41:37.096 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_pool": "true", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_standby_modules": "true", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_stats_period": "5", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_stats_threshold": "5", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_tick_period": "2", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_allow_pool_delete": "true", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T04:41:37.097 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_bytes": "104857600", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_directed_command_retry": "2", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_target_rank": "-1", 2024-09-16T04:41:37.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_file_level": "debug", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T04:41:37.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_start": "false", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_trim": "true", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cpu_threads": "4", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T04:41:37.100 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_avail_crit": "5", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_avail_warn": "5", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_json": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_extra_checks": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T04:41:37.101 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_down_uptime_grace": "60", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_election_default_strategy": "3", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_election_timeout": "5.000000", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_enable_op_tracker": "true", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_fake_pool_delete": "false", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_force_quorum_join": "false", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T04:41:37.102 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_log_update_period": "5", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_max_detail": "50", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog": "true", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_host": "172.21.15.59,172.21.15.139,172.21.15.144", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_host_override": "", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_initial_members": "", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T04:41:37.103 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease": "5.000000", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_full_interval": "50", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_max": "10000", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_max_summary": "50", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_log_epochs": "500", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_osd": "10000", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T04:41:37.104 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_memory_autotune": "true", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_memory_target": "2147483648", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_digest_period": "5", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T04:41:37.105 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_complaint_time": "30", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_duration": "600", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_size": "20", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_log_threshold": "5", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T04:41:37.106 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_cache_size": "500", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T04:41:37.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_report_timeout": "900", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T04:41:37.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T04:41:37.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_max_osds": "4", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_interval": "300", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_max_keys": "100", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_timeout": "300", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_session_timeout": "300", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_smart_report_timeout": "5", 2024-09-16T04:41:37.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_pool_size": "4", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_debug": "false", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T04:41:37.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_tick_interval": "5", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T04:41:37.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_older_version": "true", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "monmap": "", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T04:41:37.113 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_tracing_enabled": "false", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_op_threads": "3", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_cm": "false", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_device_name": "", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T04:41:37.114 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_sl": "3", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T04:41:37.115 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_type": "ib", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_reap_threshold": "5", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_before_connect": "false", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_ipv4": "true", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_ipv6": "false", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_msgr1": "true", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_msgr2": "true", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_port_max": "7568", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_port_min": "6800", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_retry_count": "3", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_retry_delay": "5", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_client": "false", 2024-09-16T04:41:37.116 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mds": "false", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mgr": "false", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mon": "false", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_osd": "false", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_client_mode": "crc secure", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_cluster_type": "", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_compress_secure": "false", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_crc_data": "true", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_crc_header": "true", 2024-09-16T04:41:37.117 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_bug": "true", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_old_message": "true", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hugepages": "", 2024-09-16T04:41:37.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_lro": "true", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_pmd": "", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_port_id": "0", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dump_on_send": "false", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_type": "", 2024-09-16T04:41:37.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_network_congestion": "0", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_socket_failures": "0", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_max_accept_failures": "4", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_max_backoff": "15.000000", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compress_mode": "none", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T04:41:37.120 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_pq_min_cost": "65536", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_public_type": "", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_service_mode": "crc secure", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_nodelay": "true", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_type": "async+posix", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "no_config_file": "false", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inflight_ops": "1024", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T04:41:37.121 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_timeout": "10.000000", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "objectstore_blackhole": "false", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "openssl_engine_opts": "", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_max_ops": "4", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_slop": "0.020000", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T04:41:37.122 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_asio_thread_count": "2", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_scan_max": "512", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_scan_min": "64", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_peering": "false", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_beacon_report_interval": "300", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_duration": "30", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T04:41:37.123 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_blkin_trace_all": "false", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_update_on_start": "true", 2024-09-16T04:41:37.124 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_message_cap": "256", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_op_priority": "63", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_watch_timeout": "30", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_command_max_records": "256", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_compact_on_start": "false", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_update_on_start": "true", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T04:41:37.125 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_op_order": "false", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T04:41:37.126 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_shutdown": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_trim_objects": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T04:41:37.127 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_default_notify_timeout": "30", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_enable_op_tracker": "true", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T04:41:37.128 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_info": "true", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown": "true", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_function_tracing": "false", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_grace": "20", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_interval": "6", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T04:41:37.129 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_stale": "600", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal_size": "5120", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_kill_backfill_at": "0", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_cache_size": "50", 2024-09-16T04:41:37.130 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_dedup": "true", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_message_max": "40", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_attr_name_len": "100", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_attr_size": "0", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_backfills": "1", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_markdown_count": "5", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_markdown_period": "600", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_name_len": "2048", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_size": "134217728", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T04:41:37.131 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pgls": "1024", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_push_cost": "8388608", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_push_objects": "10", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_scrubs": "3", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_write_size": "90", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T04:41:37.132 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_profile": "balanced", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T04:41:37.133 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_base": "805306368", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target": "4294967296", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target_autotune": "false", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_min_recovery_priority": "0", 2024-09-16T04:41:37.134 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_report_interval": "5", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_num_cache_shards": "32", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_node": "-1", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objecter_finishers": "1", 2024-09-16T04:41:37.135 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore": "bluestore", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore_fuse": "false", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore_tracing": "false", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_duration": "600", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_size": "20", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_log_threshold": "5", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards": "0", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T04:41:37.136 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_thread_timeout": "15", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_open_classes_on_start": "true", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_os_flags": "0", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_peering_op_priority": "255", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_delete_priority": "5", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T04:41:37.137 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T04:41:37.138 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_crimson": "false", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flags": "0", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_min_size": "0", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T04:41:37.139 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_size": "2", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_type": "replicated", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_cost": "20971520", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T04:41:37.140 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active": "0", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_op_priority": "3", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_priority": "5", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T04:41:37.141 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_repair_during_recovery": "false", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_cost": "52428800", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T04:41:37.142 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_end_hour": "0", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_priority": "5", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T04:41:37.143 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_skip_data_digest": "false", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_smart_report_timeout": "5", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_priority": "5", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_target_transaction_size": "30", 2024-09-16T04:41:37.144 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tracing": "false", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_use_stale_snap": "false", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T04:41:37.145 INFO:teuthology.orchestra.run.smithi059.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_kill_at": "0", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_max_join_drift": "10", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_min": "500", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_min_wait": "0.050000", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_max": "500", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_min": "250", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_stash_full_interval": "25", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_trim_max": "500", 2024-09-16T04:41:37.146 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_trim_min": "250", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "perf": "true", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "-", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrv": "", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "public_bind_addr": "-", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "public_network": "", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "public_network_interface": "", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "qat_compressor_enabled": "false", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_mon_op_timeout": "0", 2024-09-16T04:41:37.147 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_osd_op_timeout": "0", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_tracing": "false", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_atime_update_interval": "60", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache": "true", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T04:41:37.148 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_policy": "writearound", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_size": "33554432", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_compression_hint": "none", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_clone_format": "auto", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_data_pool": "", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_features": "61", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_format": "2", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_map_options": "", 2024-09-16T04:41:37.149 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_order": "22", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_pool": "rbd", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_stripe_count": "0", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_io_scheduler": "simple", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T04:41:37.150 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_order": "24", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_pool": "", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_splay_width": "4", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T04:41:37.151 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T04:41:37.152 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T04:41:37.153 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_op_threads": "1", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_plugins": "", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T04:41:37.154 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T04:41:37.155 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_tracing": "false", 2024-09-16T04:41:37.156 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_validate_names": "true", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_validate_pool": "true", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "restapi_base_url": "", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "restapi_log_level": "", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_admin_entry": "admin", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_backend_store": "rados", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_barbican_url": "", 2024-09-16T04:41:37.157 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_beast_enable_async": "true", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_enabled": "true", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_config_store": "rados", 2024-09-16T04:41:37.158 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_content_length_compat": "false", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cross_domain_policy": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T04:41:37.159 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T04:41:37.160 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T04:41:37.161 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T04:41:37.162 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d4n_port": "6379", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_window": "30", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T04:41:37.163 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T04:41:37.164 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dns_name": "", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dns_s3website_name": "", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_ops_log": "false", 2024-09-16T04:41:37.165 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_static_website": "false", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_usage_log": "false", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_expose_bucket": "false", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_extended_http_attrs": "", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_filter": "none", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T04:41:37.166 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_objs": "32", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_init_timeout": "300", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T04:41:37.167 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_password": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_project": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_token": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_user": "", 2024-09-16T04:41:37.168 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_api_version": "2", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_url": "", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T04:41:37.169 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_objs": "32", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_worker": "3", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_uri": "", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T04:41:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_http_headers": "", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attr_size": "0", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T04:41:37.171 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_listing_results": "1000", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_notify_retries": "10", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T04:41:37.172 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T04:41:37.173 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_num_control_oids": "8", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_numa_node": "-1", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_tracing": "false", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_token": "", 2024-09-16T04:41:37.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_url": "", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_rados": "false", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T04:41:37.175 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_print_continue": "true", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_tracing": "false", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm": "", 2024-09-16T04:41:37.176 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm_id": "", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_region": "", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_request_uri": "", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_resolve_cname": "false", 2024-09-16T04:41:37.177 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_run_sync_thread": "true", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_script_uri": "", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_service_provider_name": "", 2024-09-16T04:41:37.178 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_client_id": "", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_client_secret": "", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_entry": "sts", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_key": "sts", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_auth_url": "", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_custom_header": "", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_need_stats": "true", 2024-09-16T04:41:37.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_tenant_name": "", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_url": "", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_lease_period": "120", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T04:41:37.180 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_thread_pool_size": "512", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_comment": "", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_createby": "", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_encoding": "", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_flag": "false", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_origin": "", 2024-09-16T04:41:37.181 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_tracker": "", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_max_shards": "32", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_use_opa_authz": "false", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_counters_cache": "false", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T04:41:37.182 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_unique_email": "true", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_verify_ssl": "true", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone": "", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone_id": "", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup": "", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup_id": "", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T04:41:37.183 INFO:teuthology.orchestra.run.smithi059.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_block_size": "4096", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T04:41:37.184 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_partition_filters": "false", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_perf": "false", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "run_dir": "/var/run/ceph", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_block_create": "true", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T04:41:37.185 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_device_size": "53687091200", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_full_integrity_check": "false", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_main_device_type": "SSD", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T04:41:37.186 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_segment_size": "67108864", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "set_keepcaps": "false", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "setgroup": "", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "setuser": "", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "setuser_match_path": "", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "thp": "false", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "threadpool_default_timeout": "60", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T04:41:37.187 INFO:teuthology.orchestra.run.smithi059.stdout: "throttler_perf_counter": "true" 2024-09-16T04:41:37.188 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:37.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T04:41:37.336 INFO:teuthology.orchestra.run.smithi059.stdout:mon.a: {} 2024-09-16T04:41:37.337 INFO:teuthology.orchestra.run.smithi059.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:41:37.349 INFO:teuthology.orchestra.run.smithi059.stdout:mon.b: {} 2024-09-16T04:41:37.349 INFO:teuthology.orchestra.run.smithi059.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:41:37.363 INFO:teuthology.orchestra.run.smithi059.stdout:mon.c: {} 2024-09-16T04:41:37.363 INFO:teuthology.orchestra.run.smithi059.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:41:37.379 DEBUG:teuthology.orchestra.run.smithi144:> type ceph-objectstore-tool 2024-09-16T04:41:37.385 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:41:37.385 DEBUG:teuthology.orchestra.run.smithi165:> type ceph-objectstore-tool 2024-09-16T04:41:37.390 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:41:37.390 DEBUG:teuthology.orchestra.run.smithi059:> type ceph-objectstore-tool 2024-09-16T04:41:37.395 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:41:37.395 DEBUG:teuthology.orchestra.run.smithi139:> type ceph-objectstore-tool 2024-09-16T04:41:37.401 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:41:37.402 INFO:teuthology.run_tasks:Running task rados... 2024-09-16T04:41:37.413 INFO:tasks.rados:Beginning rados... 2024-09-16T04:41:37.414 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-16T04:41:37.423 INFO:tasks.rados:joining rados 2024-09-16T04:41:37.424 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T04:41:37.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:41:37.424 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:41:37.424 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T04:41:37.425 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T04:41:37.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:41:37.426 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T04:41:37.426 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:41:37.426 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T04:41:37.426 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:41:37.427 INFO:tasks.rados:clients are ['client.0'] 2024-09-16T04:41:37.427 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-09-16T04:41:37.443 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:37.440+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:37.528 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:37.521+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:37.559 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:37.559 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:37.559 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:41:37.560 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:37.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:41:37.578 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:37.579 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:37.600 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:37.601 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:37.622 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:37.622 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:37.628 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:37.629+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:37.643 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:37.644 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:37.661 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:37.662 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:37.682 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:37.683 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:37.691 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:37.692 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:37.703 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:37.704 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:37.705 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:41:37.730 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:37.724+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:37.738 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:37.756 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:37.777 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:37.777 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:37.797 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:37.818 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:37.830 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:37.829+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:37.834 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:37.834 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:37.848 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:37.849 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:37.857 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:37.857 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:37.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:41:37.879 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:37.879 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:37.902 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:37.902 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:37.965 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:37.965+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:38.004 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:38.004 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:38.017 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:41:38.061 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:38.061+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:38.134 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:38.128+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:38.181 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:38.181 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:38.195 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:41:38.235 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:38.232+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:38.318 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:38.319 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:38.331 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:41:38.336 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:38.329+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:38.437 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:38.433+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:load: shec 2024-09-16T04:41:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:41:38.461 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:38.461 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:38.481 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:41:38.482 INFO:tasks.rados:starting run 0 out of 1 2024-09-16T04:41:38.482 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T04:41:38.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-09-16T04:41:38.538 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:38.536+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:41:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:38.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:41:38.640 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:38.633+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:38.740 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:38.733+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:38.742 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:38.742 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:38.756 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:41:38.841 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:38.841+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:38.873 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:38.873 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:38.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:41:38.942 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:38.937+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:39.014 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:39.014 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:39.028 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:41:39.043 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:39.037+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:39.144 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:39.140+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:39.150 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:39.150 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:39.166 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:41:39.245 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:39.241+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:39.301 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:39.301 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:39.314 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:41:39.359 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:39.356+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:39.431 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:39.431 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:39.446 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:41:39.448 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:39.445+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:39.549 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:39.549+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:39.611 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:39.612 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:39.627 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:41:39.650 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:39.648+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:39.750 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:39.750 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:39.751 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:39.748+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:39.765 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:41:39.852 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:39.848+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:39.880 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:39.880 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:39.894 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:41:39.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:41:39.953 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:39.949+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:40.054 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:40.049+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:40.062 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:40.063 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:40.077 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:41:40.081 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:40.081 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:40.099 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:40.121 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:40.122 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:40.155 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:40.153+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:40.163 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:40.163 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:40.181 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:40.181 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:40.202 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:40.202 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:40.202 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:40.203 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:40.215 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:41:40.225 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:40.225 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:40.247 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:40.247 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:40.257 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:40.253+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:40.269 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:40.269 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:40.290 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:40.290 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:40.313 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:40.330 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:40.331 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:40.331 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:41:40.331 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:41:40.331 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:40.336 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:40.336 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:40.346 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:41:40.361 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:40.363 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:40.383 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:40.383 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:40.391 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:40.388+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:40.408 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:40.459 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:40.456+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:40.464 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:40.464 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:41:40.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:41:40.464 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [] 2024-09-16T04:41:40.464 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:40.478 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:41:40.485 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:41:40.563 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:40.560+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:40.642 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:40.642 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:40.643 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:41:40.643 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:40.657 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:41:40.661 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:40.656+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'unique_pool_0' created 2024-09-16T04:41:40.701 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T04:41:40.762 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:40.756+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:40.821 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:40.821 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:40.821 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:41:40.821 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:41:40.821 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:40.835 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:41:40.863 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:40.860+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:40.964 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:40.960+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:41.003 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:41.003 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:41:41.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.626435375, 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1755266710000001, 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.004 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374332+0000", 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8059999999999999e-06 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374345+0000", 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2529e-05 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374365+0000", 2024-09-16T04:41:41.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0695000000000001e-05 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374383+0000", 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7569000000000001e-05 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374514+0000", 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013070099999999999 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:41:41.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374544+0000", 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0731000000000001e-05 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.343896+0000", 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.969351224 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344169+0000", 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027390100000000001 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344583+0000", 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041308600000000001 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.346047+0000", 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014642699999999999 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.009 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549695+0000", 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203648463 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549791+0000", 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6137999999999993e-05 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549856+0000", 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.010 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 8.3800191959999992, 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51346886800000002, 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.011 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 111 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620747+0000", 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620757+0000", 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0316000000000001e-05 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620778+0000", 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620792+0000", 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4362000000000001e-05 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928510+0000", 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.307717717 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928629+0000", 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011919 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928822+0000", 2024-09-16T04:41:41.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019246199999999999 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930635+0000", 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00181292 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134065+0000", 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20343018399999999 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134149+0000", 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4635000000000005e-05 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134214+0000", 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4973000000000003e-05 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.017 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 8.3799488330000003, 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51366009000000001, 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 112 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620817+0000", 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:41:41.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621035+0000", 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021731100000000001 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621061+0000", 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6744999999999999e-05 2024-09-16T04:41:41.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621082+0000", 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1061999999999999e-05 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929460+0000", 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.308377346 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929595+0000", 2024-09-16T04:41:41.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013512600000000001 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929798+0000", 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020277199999999999 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932465+0000", 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002667862 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.022 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134319+0000", 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20185381599999999 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134407+0000", 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7894999999999998e-05 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134476+0000", 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8829000000000001e-05 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.023 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 8.3796488900000003, 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51357863199999998, 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.024 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 113 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:41:41.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621118+0000", 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.689e-06 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621129+0000", 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621164+0000", 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5367999999999997e-05 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621182+0000", 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8352e-05 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930689+0000", 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.309507015 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930800+0000", 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000110871 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.028 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931026+0000", 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000226293 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.933569+0000", 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002542786 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134566+0000", 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200996272 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.029 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134640+0000", 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4412000000000005e-05 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134694+0000", 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4524000000000002e-05 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.030 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 8.3795521429999997, 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51372108500000002, 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 114 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:41:41.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621214+0000", 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621225+0000", 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1348999999999999e-05 2024-09-16T04:41:41.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621245+0000", 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9966000000000002e-05 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621259+0000", 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4102e-05 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.034 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931642+0000", 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31038274300000002 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931776+0000", 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133552 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931973+0000", 2024-09-16T04:41:41.035 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197262 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.033893+0000", 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101919751 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134774+0000", 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10088169600000001 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134847+0000", 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2309000000000002e-05 2024-09-16T04:41:41.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134934+0000", 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6924999999999999e-05 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 8.3794803340000001, 2024-09-16T04:41:41.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51387291300000004, 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 115 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621286+0000", 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.26e-06 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.040 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621492+0000", 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020643500000000001 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621517+0000", 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621534+0000", 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6515999999999999e-05 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.041 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932515+0000", 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31098146799999998 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932631+0000", 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011636 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932814+0000", 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000182406 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134236+0000", 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201422618 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135029+0000", 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079253199999999998 2024-09-16T04:41:41.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135102+0000", 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3286000000000003e-05 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135157+0000", 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5043000000000001e-05 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 8.3791985160000007, 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.717172162, 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.045 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 116 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.046 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621568+0000", 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621606+0000", 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8167999999999998e-05 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621628+0000", 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1742e-05 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621643+0000", 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5696e-05 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235313+0000", 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61366968300000002 2024-09-16T04:41:41.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235449+0000", 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136036 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235657+0000", 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207405 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237049+0000", 2024-09-16T04:41:41.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001392552 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338582+0000", 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101532626 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338670+0000", 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7888999999999998e-05 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338738+0000", 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8769000000000005e-05 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 8.3790905260000006, 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.81671015999999996, 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.051 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 117 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.052 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621676+0000", 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4440000000000001e-06 2024-09-16T04:41:41.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621686+0000", 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0845e-05 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621706+0000", 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9168999999999999e-05 2024-09-16T04:41:41.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621723+0000", 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7538999999999999e-05 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236241+0000", 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61451815600000004 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236372+0000", 2024-09-16T04:41:41.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013062300000000001 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236574+0000", 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020165600000000001 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237570+0000", 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099642999999999997 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.056 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438224+0000", 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20065369 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438317+0000", 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3624999999999999e-05 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438384+0000", 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6983000000000006e-05 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294489+0000", 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.7062757609999997, 2024-09-16T04:41:41.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.50794601399999995, 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 137 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294489+0000", 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294489+0000", 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294491+0000", 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.66e-06 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294670+0000", 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017951999999999999 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294695+0000", 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5315000000000001e-05 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294716+0000", 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0194999999999999e-05 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.403886+0000", 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.109170657 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.404000+0000", 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011403 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.404180+0000", 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000179624 2024-09-16T04:41:41.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709539+0000", 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30535920900000002 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802235+0000", 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092695601000000002 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802323+0000", 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8059000000000005e-05 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802435+0000", 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000112144 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.065 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.7060165490000001, 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.508075677, 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 138 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294750+0000", 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:41:41.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294760+0000", 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294779+0000", 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8941999999999999e-05 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294794+0000", 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5038999999999999e-05 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405306+0000", 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11051159300000001 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405426+0000", 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011977 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405611+0000", 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000185217 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709561+0000", 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394976699999998 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802549+0000", 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092988137999999998 2024-09-16T04:41:41.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802707+0000", 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015861799999999999 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802824+0000", 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116517 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.072 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.072 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:41:41.072 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.7059464259999997, 2024-09-16T04:41:41.072 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60756662299999997, 2024-09-16T04:41:41.072 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.072 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 139 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294820+0000", 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4330000000000001e-06 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294830+0000", 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0023e-05 2024-09-16T04:41:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294850+0000", 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0622e-05 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294864+0000", 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4042e-05 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406110+0000", 2024-09-16T04:41:41.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11124526 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406222+0000", 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111871 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406398+0000", 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017598000000000001 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709577+0000", 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30317971500000002 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902101+0000", 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19252423499999999 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902267+0000", 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016537499999999999 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902385+0000", 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011806700000000001 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.7058542980000002, 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60781196900000001, 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.078 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 140 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294912+0000", 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.407e-06 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295040+0000", 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012836600000000001 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295062+0000", 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1682e-05 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295083+0000", 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1109000000000001e-05 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507008+0000", 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21192486699999999 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507149+0000", 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141042 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507369+0000", 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022046500000000001 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710298+0000", 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202928308 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902525+0000", 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.192226958 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902611+0000", 2024-09-16T04:41:41.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6274999999999996e-05 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902722+0000", 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011149 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:41:41.084 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.7056267170000003, 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70811914799999998, 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 141 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:41:41.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295140+0000", 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295152+0000", 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2053e-05 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295172+0000", 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0282e-05 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295191+0000", 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8759000000000001e-05 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508486+0000", 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.213295545 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508630+0000", 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014419899999999999 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508824+0000", 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019358600000000001 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710318+0000", 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20149381 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.002981+0000", 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.29266302500000002 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003144+0000", 2024-09-16T04:41:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016330500000000001 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003257+0000", 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113064 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:41:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0411699890000001, 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51415646199999998, 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 159 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959597+0000", 2024-09-16T04:41:41.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9190000000000002e-06 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959609+0000", 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959629+0000", 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0035e-05 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959645+0000", 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5786000000000001e-05 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167529+0000", 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.207883768 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167762+0000", 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023295499999999999 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.168134+0000", 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037181099999999999 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.270830+0000", 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102696683 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473585+0000", 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20275470700000001 2024-09-16T04:41:41.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473677+0000", 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1772000000000002e-05 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473751+0000", 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4228999999999997e-05 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0410892269999996, 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51420228400000001, 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 160 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.098 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959677+0000", 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:41:41.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959687+0000", 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959711+0000", 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3811999999999998e-05 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959727+0000", 2024-09-16T04:41:41.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169193+0000", 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209466241 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169340+0000", 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000147279 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169545+0000", 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000205167 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374117+0000", 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20457175999999999 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473763+0000", 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099646306000000004 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473828+0000", 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4046000000000003e-05 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473878+0000", 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0241000000000002e-05 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0410119480000004, 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51424688600000001, 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 161 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959754+0000", 2024-09-16T04:41:41.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959920+0000", 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166088 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959944+0000", 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959963+0000", 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9176000000000001e-05 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371049+0000", 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41108644500000002 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371185+0000", 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013619899999999999 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371395+0000", 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020912 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.377708+0000", 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006313173 2024-09-16T04:41:41.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473891+0000", 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.096182904 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473949+0000", 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8016999999999997e-05 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474000+0000", 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0406961910000003, 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51552797100000003, 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.111 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 162 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960070+0000", 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960077+0000", 2024-09-16T04:41:41.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6830000000000008e-06 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960091+0000", 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3653e-05 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960101+0000", 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7629999999999995e-06 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372554+0000", 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41245356599999999 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372785+0000", 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023023100000000001 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.373156+0000", 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000371601 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474010+0000", 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100854291 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475328+0000", 2024-09-16T04:41:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001317304 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475477+0000", 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014954899999999999 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475596+0000", 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011913699999999999 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0406479629999996, 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51788103100000005, 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.118 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 163 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960118+0000", 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960124+0000", 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960134+0000", 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0115999999999999e-05 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960144+0000", 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374237+0000", 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41409377600000002 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374462+0000", 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022478600000000001 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374797+0000", 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000334586 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474018+0000", 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099220947000000004 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477813+0000", 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003795171 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477921+0000", 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010822100000000001 2024-09-16T04:41:41.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477998+0000", 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6662999999999994e-05 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.124 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0406087490000004, 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51822496799999995, 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 164 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960157+0000", 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7399999999999991e-07 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960241+0000", 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3677999999999994e-05 2024-09-16T04:41:41.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960256+0000", 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5206e-05 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960267+0000", 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1372999999999999e-05 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375650+0000", 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41538323100000002 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375885+0000", 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000234758 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.376293+0000", 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040777599999999999 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474024+0000", 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097731440000000003 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478113+0000", 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004088225 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478261+0000", 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014825 2024-09-16T04:41:41.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478381+0000", 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012005700000000001 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:41.131 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0404826759999999, 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51862660500000002, 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:41.132 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 165 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960283+0000", 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960289+0000", 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.49e-06 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960301+0000", 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2585999999999999e-05 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960309+0000", 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472635+0000", 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51232596699999999 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472771+0000", 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136243 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473033+0000", 2024-09-16T04:41:41.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026155699999999999 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474539+0000", 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015065 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.137 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478656+0000", 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041164319999999997 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478793+0000", 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013757299999999999 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478909+0000", 2024-09-16T04:41:41.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011530400000000001 2024-09-16T04:41:41.139 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.139 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.139 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.139 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:41.139 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:41.139 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:41.140 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:41:41.141 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:41.136+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:41.242 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:41.240+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:41:41.273 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:41.273 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:41.273 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:41:41.273 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:41.287 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:41:41.343 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:41.336+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:41.411 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:41.412 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:41.412 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:41.412 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:41.412 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:41.426 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:41:41.444 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:41.441+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:41.544 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:41.537+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:41.552 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:42 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'28, mlcod=26'26)", 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:28.043833+0000", 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.506126646, 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.201486586, 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.043833+0000", 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.043833+0000", 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.043835+0000", 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6559999999999999e-06 2024-09-16T04:41:41.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.043847+0000", 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1805e-05 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.043875+0000", 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8345000000000001e-05 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.143969+0000", 2024-09-16T04:41:41.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100093585 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.244232+0000", 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100263319 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.244265+0000", 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2604000000000002e-05 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.245233+0000", 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00096869000000000002 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:28.245320+0000", 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6582000000000006e-05 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.557 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:69 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'43, mlcod=27'39)", 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:30.097036+0000", 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.452924346, 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14866917900000001, 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.097036+0000", 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.097036+0000", 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.097036+0000", 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7700000000000004e-07 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.097094+0000", 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7318000000000002e-05 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.097106+0000", 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2302e-05 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.097115+0000", 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8689999999999993e-06 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.244065+0000", 2024-09-16T04:41:41.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.146949728 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.244098+0000", 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3303999999999998e-05 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.245642+0000", 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001543607 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.245705+0000", 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3373999999999993e-05 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:70 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'44, mlcod=27'39)", 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:30.099529+0000", 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.450431558, 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14690081899999999, 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.562 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.099529+0000", 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.099529+0000", 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.099530+0000", 2024-09-16T04:41:41.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6160000000000001e-06 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.099540+0000", 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0193e-05 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.099568+0000", 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7971e-05 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.199705+0000", 2024-09-16T04:41:41.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100136185 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.245109+0000", 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.045404462999999999 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.245168+0000", 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9385000000000001e-05 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.246341+0000", 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011723969999999999 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:30.246429+0000", 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8609000000000005e-05 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:119 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'81, mlcod=28'78)", 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:41:41.566 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.1050706199999993, 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20259546, 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444891+0000", 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6360000000000001e-06 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.445043+0000", 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000151581 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.445078+0000", 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5077000000000003e-05 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.545224+0000", 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10014579899999999 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.645477+0000", 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100253114 2024-09-16T04:41:41.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.645521+0000", 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4218999999999998e-05 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.647416+0000", 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001895477 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.647485+0000", 2024-09-16T04:41:41.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8557000000000001e-05 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:133 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'91, mlcod=28'86)", 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:34.401593+0000", 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 7.148366835, 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20160451500000001, 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.571 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.401593+0000", 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.401593+0000", 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.401595+0000", 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8899999999999999e-06 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.401611+0000", 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5682e-05 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.401655+0000", 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4299999999999999e-05 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.401669+0000", 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3952e-05 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.499911+0000", 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098241588000000005 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.499964+0000", 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3133000000000001e-05 2024-09-16T04:41:41.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603106+0000", 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10314208 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603198+0000", 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1890000000000003e-05 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:134 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'92, mlcod=28'86)", 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:34.402573+0000", 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 7.1473870159999997, 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20069947299999999, 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.402573+0000", 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.402573+0000", 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.402575+0000", 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.733e-06 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.402631+0000", 2024-09-16T04:41:41.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6171999999999997e-05 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.402660+0000", 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8894000000000002e-05 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.402674+0000", 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3638999999999999e-05 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.500480+0000", 2024-09-16T04:41:41.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097806039999999997 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.500508+0000", 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7937e-05 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603206+0000", 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10269858 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603273+0000", 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6477999999999996e-05 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:135 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'93, mlcod=28'86)", 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:34.403707+0000", 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 7.1462535760000003, 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19963573400000001, 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.580 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.403707+0000", 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.403707+0000", 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.403708+0000", 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6929999999999999e-06 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.403720+0000", 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1319e-05 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.403748+0000", 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8061000000000001e-05 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.403761+0000", 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3549e-05 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.501108+0000", 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097346928999999999 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.501178+0000", 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9690999999999994e-05 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603279+0000", 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102101021 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603342+0000", 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3471000000000003e-05 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:136 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'94, mlcod=28'86)", 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:34.404597+0000", 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 7.1453635709999999, 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.201207409, 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.404597+0000", 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.404597+0000", 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.404598+0000", 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.857e-06 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.404611+0000", 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2119e-05 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.404639+0000", 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8444000000000001e-05 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.404653+0000", 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3619e-05 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.601165+0000", 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19651269399999999 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.601198+0000", 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2323999999999997e-05 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.605721+0000", 2024-09-16T04:41:41.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0045237760000000002 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.605804+0000", 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2576000000000004e-05 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:137 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'95, mlcod=28'86)", 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:34.405479+0000", 2024-09-16T04:41:41.589 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 7.1444807580000003, 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040242, 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.405479+0000", 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.405479+0000", 2024-09-16T04:41:41.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.405482+0000", 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7599999999999998e-06 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.405546+0000", 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3894999999999997e-05 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.405576+0000", 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9937000000000002e-05 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.405590+0000", 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3813e-05 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.601827+0000", 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19623687500000001 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.592 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.601876+0000", 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8943000000000002e-05 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.605811+0000", 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0039353560000000001 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.605882+0000", 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0840999999999995e-05 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.593 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:138 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'96, mlcod=28'86)", 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:34.406865+0000", 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 7.1430952249999997, 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19907998599999999, 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.406865+0000", 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.406865+0000", 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.406867+0000", 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6869999999999999e-06 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.406878+0000", 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1175000000000001e-05 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.406905+0000", 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7725e-05 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.406919+0000", 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3692e-05 2024-09-16T04:41:41.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603436+0000", 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19651713700000001 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603476+0000", 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9397000000000002e-05 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.605884+0000", 2024-09-16T04:41:41.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024084940000000002 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.605945+0000", 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0679e-05 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:139 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'97, mlcod=28'86)", 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:34.407572+0000", 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 7.1423884119999999, 2024-09-16T04:41:41.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19844019399999999, 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.407572+0000", 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.407572+0000", 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.407573+0000", 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.677e-06 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.407585+0000", 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1436e-05 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.407613+0000", 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7827e-05 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.407626+0000", 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3549e-05 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603948+0000", 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.196322043 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.603985+0000", 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6529999999999998e-05 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.605949+0000", 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001964604 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:34.606012+0000", 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2527999999999995e-05 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.602 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:146 1.0 e29/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 29'104, mlcod=28'102)", 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.1362559020000003, 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20228226799999999, 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413706+0000", 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7400000000000001e-06 2024-09-16T04:41:41.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413718+0000", 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1892000000000001e-05 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413749+0000", 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0848e-05 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.513883+0000", 2024-09-16T04:41:41.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100134561 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.614137+0000", 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100254093 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.614197+0000", 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9703000000000001e-05 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.615867+0000", 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001670073 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.615987+0000", 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000119358 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:168 1.0 e29/23 1:30306672:devicehealth::main.db.0000000000000000:head v 29'122, mlcod=29'119)", 2024-09-16T04:41:41.607 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:36.682449+0000", 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.8675114050000001, 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20211510199999999, 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682449+0000", 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682449+0000", 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682451+0000", 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.846e-06 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682570+0000", 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011917399999999999 2024-09-16T04:41:41.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682599+0000", 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9328000000000001e-05 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.782753+0000", 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100153699 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.782936+0000", 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000183606 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.782983+0000", 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6536999999999998e-05 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.884482+0000", 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101499362 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.884564+0000", 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1550000000000004e-05 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:202 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'148, mlcod=29'145)", 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.2036892809999999, 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30312661499999999, 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.612 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346273+0000", 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7630000000000001e-06 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346284+0000", 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1489e-05 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346311+0000", 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6888000000000001e-05 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346325+0000", 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3669000000000001e-05 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.445851+0000", 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099526192999999999 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.445885+0000", 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4075000000000002e-05 2024-09-16T04:41:41.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649211+0000", 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20332629399999999 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649397+0000", 2024-09-16T04:41:41.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018624399999999999 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:203 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'149, mlcod=29'145)", 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.2032211079999999, 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30271984099999999, 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.617 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346740+0000", 2024-09-16T04:41:41.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0699999999999996e-07 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346874+0000", 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000134585 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346885+0000", 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0283e-05 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.446978+0000", 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009309199999999 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447153+0000", 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000174721 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447187+0000", 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4913999999999998e-05 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649408+0000", 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20222030199999999 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649459+0000", 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1137000000000002e-05 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.621 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:204 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'150, mlcod=29'145)", 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.102907796, 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246620800000001, 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447054+0000", 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3430000000000001e-06 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447066+0000", 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2055999999999999e-05 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447085+0000", 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9165e-05 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447099+0000", 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3791999999999999e-05 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448012+0000", 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000913319 2024-09-16T04:41:41.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448042+0000", 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0074e-05 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649463+0000", 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20142125599999999 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649519+0000", 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5203e-05 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:205 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'151, mlcod=29'145)", 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.1028433339999999, 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202457419, 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.626 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447123+0000", 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5600000000000001e-06 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447133+0000", 2024-09-16T04:41:41.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7249999999999992e-06 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447140+0000", 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.204e-06 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448504+0000", 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013644340000000001 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448533+0000", 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8691e-05 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649523+0000", 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20098977900000001 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649574+0000", 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1523000000000001e-05 2024-09-16T04:41:41.630 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:206 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'152, mlcod=29'145)", 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.102810233, 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20247812800000001, 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:41.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.03e-07 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447206+0000", 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5952999999999998e-05 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447217+0000", 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0533000000000001e-05 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447228+0000", 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1012000000000001e-05 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448968+0000", 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001740412 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448996+0000", 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7331999999999999e-05 2024-09-16T04:41:41.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649579+0000", 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20058309599999999 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649628+0000", 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9387e-05 2024-09-16T04:41:41.635 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:207 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'153, mlcod=29'145)", 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447828+0000", 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.102132213, 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20223788500000001, 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447828+0000", 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447828+0000", 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447829+0000", 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.184e-06 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447838+0000", 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447853+0000", 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5189e-05 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447863+0000", 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0229e-05 2024-09-16T04:41:41.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.648746+0000", 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20088268300000001 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.648779+0000", 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3104e-05 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.650016+0000", 2024-09-16T04:41:41.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00123682 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.650066+0000", 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0049999999999997e-05 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:209 1.0 e32/23 1:30306672:devicehealth::main.db.0000000000000000:head v 32'155, mlcod=30'152)", 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:41:41.640 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.79724513600000002, 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20243324500000001, 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:41:41.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752717+0000", 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.66e-06 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752778+0000", 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1816000000000006e-05 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752805+0000", 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6335000000000001e-05 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.852941+0000", 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100135847 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:41.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.953240+0000", 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10029893300000001 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.953275+0000", 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5091e-05 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.955059+0000", 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001784683 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:41.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.955148+0000", 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8880000000000003e-05 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:41.645 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:41.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:41:41.647 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:41.645+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:41:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T04:41:41.747 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:41.744+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:41.816 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:41.817 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:41.817 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:41:41.817 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:41.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:41:41.877 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:41.877+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:41.948 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:41.944+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:41.958 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:41.958 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:41.958 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:41.958 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:41.958 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:41.973 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:41:42.049 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:42.049+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:42.140 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:42.141 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:41:42.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:41:42.141 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [] 2024-09-16T04:41:42.141 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:42.150 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:42.144+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:42.155 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:41:42.251 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:42.249+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:42.277 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:42.277 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:41:42.277 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:42.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:41:42.353 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:42.348+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:42.413 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:42.413 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:42.413 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:42.413 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:42.413 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:42.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:41:42.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:41:42.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:41:42.430 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:41:42.477 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:42.477+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:42.549 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:42.549 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:41:42.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:41:42.550 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [] 2024-09-16T04:41:42.550 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:42.554 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:42.553+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:42.563 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:41:42.589 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:42.589 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:42.611 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:42.611 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:42.629 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:42.630 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:42.650 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:42.651 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:42.655 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:42.652+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:42.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:42.668 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:42.683 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:42.683 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:42.683 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:41:42.683 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:42.689 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:42.689 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:42.697 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:41:42.709 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:42.709 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:42.731 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:42.732 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:42.741 DEBUG:teuthology.orchestra.run.smithi165:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-16T04:41:42.742 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T04:41:42.742 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:42.753 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:42.753 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:42.756 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:42.752+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:42.763 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight read -> 100 2024-09-16T04:41:42.763 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight delete -> 50 2024-09-16T04:41:42.763 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight snap_create -> 50 2024-09-16T04:41:42.763 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight snap_remove -> 50 2024-09-16T04:41:42.763 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight rollback -> 50 2024-09-16T04:41:42.763 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight setattr -> 25 2024-09-16T04:41:42.763 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight rmattr -> 25 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight copy_from -> 50 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight append -> 100 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout:ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout:Configuration: 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout: Number of operations: 400 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout: Number of objects: 50 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout: Max in flight operations: 16 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout: Object size (in bytes): 4000000 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout: Write stride min: 400000 2024-09-16T04:41:42.764 INFO:tasks.rados.rados.0.smithi165.stdout: Write stride max: 800000 2024-09-16T04:41:42.776 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:42.797 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:42.797 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:42.819 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:42.819 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:42.823 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:42.823 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:42.837 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:41:42.838 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:42.860 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:42.860 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:42.880 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:42.880 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:42.881 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:42.881+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:42.901 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:42.901 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:42.955 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:42.955 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:42.961 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:42.961+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:42.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:41:43.046 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:43.047 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:41:43.059 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:43.056+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:43.066 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-09-16T04:41:42.990485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":134,"num_read_kb":116,"num_write":200,"num_write_kb":3333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437284,"kb_used_data":4808,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512156,"statfs":{"total":1546188226560,"available":1545740447744,"internally_reserved":0,"allocated":4923392,"data_stored":3227369,"data_compressed":424907,"data_compressed_allocated":606208,"data_compressed_original":2019328,"omap_allocated":25522,"internal_metadata":442473550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":27,"num_write":56,"num_write_kb":1178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001296"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:41:42.154735+0000","last_change":"2024-09-16T04:41:42.153067+0000","last_active":"2024-09-16T04:41:42.154735+0000","last_peered":"2024-09-16T04:41:42.154735+0000","last_clean":"2024-09-16T04:41:42.154735+0000","last_became_active":"2024-09-16T04:41:41.950892+0000","last_became_peered":"2024-09-16T04:41:41.950892+0000","last_unstale":"2024-09-16T04:41:42.154735+0000","last_undegraded":"2024-09-16T04:41:42.154735+0000","last_fullsized":"2024-09-16T04:41:42.154735+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:10.104988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:41:42.060863+0000","last_change":"2024-09-16T04:41:42.059491+0000","last_active":"2024-09-16T04:41:42.060863+0000","last_peered":"2024-09-16T04:41:42.060863+0000","last_clean":"2024-09-16T04:41:42.060863+0000","last_became_active":"2024-09-16T04:41:42.057674+0000","last_became_peered":"2024-09-16T04:41:42.057674+0000","last_unstale":"2024-09-16T04:41:42.060863+0000","last_undegraded":"2024-09-16T04:41:42.060863+0000","last_fullsized":"2024-09-16T04:41:42.060863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:59.753480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"30'153","reported_seq":208,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:40.342670+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:41:40.342670+0000","last_peered":"2024-09-16T04:41:40.342670+0000","last_clean":"2024-09-16T04:41:40.342670+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:41:40.342670+0000","last_undegraded":"2024-09-16T04:41:40.342670+0000","last_fullsized":"2024-09-16T04:41:40.342670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":134,"num_read_kb":116,"num_write":198,"num_write_kb":3331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.559846+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:38.559846+0000","last_peered":"2024-09-16T04:41:38.559846+0000","last_clean":"2024-09-16T04:41:38.559846+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:38.559846+0000","last_undegraded":"2024-09-16T04:41:38.559846+0000","last_fullsized":"2024-09-16T04:41:38.559846+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.801676+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:38.801676+0000","last_peered":"2024-09-16T04:41:38.801676+0000","last_clean":"2024-09-16T04:41:38.801676+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:38.801676+0000","last_undegraded":"2024-09-16T04:41:38.801676+0000","last_fullsized":"2024-09-16T04:41:38.801676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.663123+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:41:38.663123+0000","last_peered":"2024-09-16T04:41:38.663123+0000","last_clean":"2024-09-16T04:41:38.663123+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:41:38.663123+0000","last_undegraded":"2024-09-16T04:41:38.663123+0000","last_fullsized":"2024-09-16T04:41:38.663123+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.801706+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:38.801706+0000","last_peered":"2024-09-16T04:41:38.801706+0000","last_clean":"2024-09-16T04:41:38.801706+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:38.801706+0000","last_undegraded":"2024-09-16T04:41:38.801706+0000","last_fullsized":"2024-09-16T04:41:38.801706+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.702306+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:38.702306+0000","last_peered":"2024-09-16T04:41:38.702306+0000","last_clean":"2024-09-16T04:41:38.702306+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:38.702306+0000","last_undegraded":"2024-09-16T04:41:38.702306+0000","last_fullsized":"2024-09-16T04:41:38.702306+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.701598+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:38.701598+0000","last_peered":"2024-09-16T04:41:38.701598+0000","last_clean":"2024-09-16T04:41:38.701598+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:38.701598+0000","last_undegraded":"2024-09-16T04:41:38.701598+0000","last_fullsized":"2024-09-16T04:41:38.701598+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:40.675693+0000","last_change":"2024-09-16T04:41:40.675693+0000","last_active":"2024-09-16T04:41:40.675693+0000","last_peered":"2024-09-16T04:41:40.675693+0000","last_clean":"2024-09-16T04:41:40.675693+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:40.675693+0000","last_undegraded":"2024-09-16T04:41:40.675693+0000","last_fullsized":"2024-09-16T04:41:40.675693+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.573953+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:38.573953+0000","last_peered":"2024-09-16T04:41:38.573953+0000","last_clean":"2024-09-16T04:41:38.573953+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:38.573953+0000","last_undegraded":"2024-09-16T04:41:38.573953+0000","last_fullsized":"2024-09-16T04:41:38.573953+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T04:41:42.065378+0000","last_change":"2024-09-16T04:41:42.063184+0000","last_active":"2024-09-16T04:41:42.065378+0000","last_peered":"2024-09-16T04:41:42.065378+0000","last_clean":"2024-09-16T04:41:42.065378+0000","last_became_active":"2024-09-16T04:41:42.061226+0000","last_became_peered":"2024-09-16T04:41:42.061226+0000","last_unstale":"2024-09-16T04:41:42.065378+0000","last_undegraded":"2024-09-16T04:41:42.065378+0000","last_fullsized":"2024-09-16T04:41:42.065378+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.598744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T04:41:38.801212+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:38.801212+0000","last_peered":"2024-09-16T04:41:38.801212+0000","last_clean":"2024-09-16T04:41:38.801212+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:38.801212+0000","last_undegraded":"2024-09-16T04:41:38.801212+0000","last_fullsized":"2024-09-16T04:41:38.801212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":21,"acting":21,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":134,"num_read_kb":116,"num_write":198,"num_write_kb":3331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":996416,"data_compressed":60898,"data_compressed_allocated":77824,"data_compressed_original":962560,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":667468,"data_compressed":54472,"data_compressed_allocated":73728,"data_compressed_original":577536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247816,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247816,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":159851,"data_compressed":28394,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":335872,"data_stored":170152,"data_compressed":31311,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280519,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":344064,"data_stored":596366,"data_compressed":48665,"data_compressed_allocated":65536,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":128897,"data_compressed":19681,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":128878,"data_compressed":19681,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":138540,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":303104,"data_stored":138559,"data_compressed":22558,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":128878,"data_compressed":19681,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":28984,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31914,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:43.068 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T04:41:43.068 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-16T04:41:43.103 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:43.103 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:43.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:41:43.162 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:43.160+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:43.260 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:43.261+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:43.285 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:43.285 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:43.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:41:43.362 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:43.356+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:43.420 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:43.420 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:43.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:41:43.463 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:43.461+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:43.558 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:43.558 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:43.563 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:43.560+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:43.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:41:43.664 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:43.661+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:43.690 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:43.690 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:43.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:41:43.708 INFO:tasks.rados.rados.0.smithi165.stdout:1: write initial oid 1 2024-09-16T04:41:43.708 INFO:tasks.rados.rados.0.smithi165.stdout:1: seq_num 0 ranges {0=32768} 2024-09-16T04:41:43.708 INFO:tasks.rados.rados.0.smithi165.stdout:1: writing smithi16530038-1 from 0 to 32768 tid 1 2024-09-16T04:41:43.708 INFO:tasks.rados.rados.0.smithi165.stdout:2: write initial oid 2 2024-09-16T04:41:43.708 INFO:tasks.rados.rados.0.smithi165.stdout:2: seq_num 1 ranges {0=32768} 2024-09-16T04:41:43.709 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pg_num to 26 2024-09-16T04:41:43.709 INFO:tasks.rados.rados.0.smithi165.stdout:2: writing smithi16530038-2 from 0 to 32768 tid 1 2024-09-16T04:41:43.709 INFO:tasks.rados.rados.0.smithi165.stdout:3: write initial oid 3 2024-09-16T04:41:43.710 INFO:tasks.rados.rados.0.smithi165.stdout:3: seq_num 2 ranges {0=49152} 2024-09-16T04:41:43.710 INFO:tasks.rados.rados.0.smithi165.stdout:3: writing smithi16530038-3 from 0 to 49152 tid 1 2024-09-16T04:41:43.710 INFO:tasks.rados.rados.0.smithi165.stdout:4: write initial oid 4 2024-09-16T04:41:43.710 INFO:tasks.rados.rados.0.smithi165.stdout:4: seq_num 3 ranges {0=65536} 2024-09-16T04:41:43.710 INFO:tasks.rados.rados.0.smithi165.stdout:4: writing smithi16530038-4 from 0 to 65536 tid 1 2024-09-16T04:41:43.710 INFO:tasks.rados.rados.0.smithi165.stdout:5: write initial oid 5 2024-09-16T04:41:43.710 INFO:tasks.rados.rados.0.smithi165.stdout:5: seq_num 4 ranges {0=81920} 2024-09-16T04:41:43.711 INFO:tasks.rados.rados.0.smithi165.stdout:5: writing smithi16530038-5 from 0 to 81920 tid 1 2024-09-16T04:41:43.711 INFO:tasks.rados.rados.0.smithi165.stdout:6: write initial oid 6 2024-09-16T04:41:43.711 INFO:tasks.rados.rados.0.smithi165.stdout:6: seq_num 5 ranges {0=98304} 2024-09-16T04:41:43.712 INFO:tasks.rados.rados.0.smithi165.stdout:6: writing smithi16530038-6 from 0 to 98304 tid 1 2024-09-16T04:41:43.712 INFO:tasks.rados.rados.0.smithi165.stdout:7: write initial oid 7 2024-09-16T04:41:43.712 INFO:tasks.rados.rados.0.smithi165.stdout:7: seq_num 6 ranges {0=114688} 2024-09-16T04:41:43.714 INFO:tasks.rados.rados.0.smithi165.stdout:7: writing smithi16530038-7 from 0 to 114688 tid 1 2024-09-16T04:41:43.714 INFO:tasks.rados.rados.0.smithi165.stdout:8: write initial oid 8 2024-09-16T04:41:43.714 INFO:tasks.rados.rados.0.smithi165.stdout:8: seq_num 7 ranges {0=131072} 2024-09-16T04:41:43.715 INFO:tasks.rados.rados.0.smithi165.stdout:8: writing smithi16530038-8 from 0 to 131072 tid 1 2024-09-16T04:41:43.715 INFO:tasks.rados.rados.0.smithi165.stdout:9: write initial oid 9 2024-09-16T04:41:43.715 INFO:tasks.rados.rados.0.smithi165.stdout:9: seq_num 8 ranges {0=147456} 2024-09-16T04:41:43.717 INFO:tasks.rados.rados.0.smithi165.stdout:9: writing smithi16530038-9 from 0 to 147456 tid 1 2024-09-16T04:41:43.717 INFO:tasks.rados.rados.0.smithi165.stdout:10: write initial oid 10 2024-09-16T04:41:43.717 INFO:tasks.rados.rados.0.smithi165.stdout:10: seq_num 9 ranges {0=163840} 2024-09-16T04:41:43.718 INFO:tasks.rados.rados.0.smithi165.stdout:10: writing smithi16530038-10 from 0 to 163840 tid 1 2024-09-16T04:41:43.718 INFO:tasks.rados.rados.0.smithi165.stdout:11: write initial oid 11 2024-09-16T04:41:43.718 INFO:tasks.rados.rados.0.smithi165.stdout:11: seq_num 10 ranges {0=180224} 2024-09-16T04:41:43.720 INFO:tasks.rados.rados.0.smithi165.stdout:11: writing smithi16530038-11 from 0 to 180224 tid 1 2024-09-16T04:41:43.720 INFO:tasks.rados.rados.0.smithi165.stdout:12: write initial oid 12 2024-09-16T04:41:43.720 INFO:tasks.rados.rados.0.smithi165.stdout:12: seq_num 11 ranges {0=196608} 2024-09-16T04:41:43.721 INFO:tasks.rados.rados.0.smithi165.stdout:12: writing smithi16530038-12 from 0 to 196608 tid 1 2024-09-16T04:41:43.722 INFO:tasks.rados.rados.0.smithi165.stdout:13: write initial oid 13 2024-09-16T04:41:43.722 INFO:tasks.rados.rados.0.smithi165.stdout:13: seq_num 12 ranges {0=212992} 2024-09-16T04:41:43.723 INFO:tasks.rados.rados.0.smithi165.stdout:13: writing smithi16530038-13 from 0 to 212992 tid 1 2024-09-16T04:41:43.723 INFO:tasks.rados.rados.0.smithi165.stdout:14: write initial oid 14 2024-09-16T04:41:43.723 INFO:tasks.rados.rados.0.smithi165.stdout:14: seq_num 13 ranges {0=229376} 2024-09-16T04:41:43.725 INFO:tasks.rados.rados.0.smithi165.stdout:14: writing smithi16530038-14 from 0 to 229376 tid 1 2024-09-16T04:41:43.725 INFO:tasks.rados.rados.0.smithi165.stdout:15: write initial oid 15 2024-09-16T04:41:43.725 INFO:tasks.rados.rados.0.smithi165.stdout:15: seq_num 14 ranges {0=245760} 2024-09-16T04:41:43.728 INFO:tasks.rados.rados.0.smithi165.stdout:15: writing smithi16530038-15 from 0 to 245760 tid 1 2024-09-16T04:41:43.728 INFO:tasks.rados.rados.0.smithi165.stdout:16: write initial oid 16 2024-09-16T04:41:43.728 INFO:tasks.rados.rados.0.smithi165.stdout:16: seq_num 15 ranges {0=262144} 2024-09-16T04:41:43.729 INFO:tasks.rados.rados.0.smithi165.stdout:16: writing smithi16530038-16 from 0 to 262144 tid 1 2024-09-16T04:41:43.730 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:43.740 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:41:43.765 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:43.760+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:43.819 INFO:tasks.rados.rados.0.smithi165.stdout:4: finishing write tid 1 to smithi16530038-4 2024-09-16T04:41:43.837 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:43.837 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:43.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:41:43.894 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:43.892+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:43.967 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:43.965+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:44.023 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:44.023 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:44.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:41:44.068 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:44.064+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:44.122 INFO:tasks.rados.rados.0.smithi165.stdout:6: finishing write tid 1 to smithi16530038-6 2024-09-16T04:41:44.122 INFO:tasks.rados.rados.0.smithi165.stdout:6: finishing write tid 2 to smithi16530038-6 2024-09-16T04:41:44.128 INFO:tasks.rados.rados.0.smithi165.stdout:8: finishing write tid 1 to smithi16530038-8 2024-09-16T04:41:44.129 INFO:tasks.rados.rados.0.smithi165.stdout:4: finishing write tid 2 to smithi16530038-4 2024-09-16T04:41:44.131 INFO:tasks.rados.rados.0.smithi165.stdout:7: finishing write tid 1 to smithi16530038-7 2024-09-16T04:41:44.131 INFO:tasks.rados.rados.0.smithi165.stdout:7: finishing write tid 2 to smithi16530038-7 2024-09-16T04:41:44.133 INFO:tasks.rados.rados.0.smithi165.stdout:3: finishing write tid 1 to smithi16530038-3 2024-09-16T04:41:44.133 INFO:tasks.rados.rados.0.smithi165.stdout:3: finishing write tid 2 to smithi16530038-3 2024-09-16T04:41:44.133 INFO:tasks.rados.rados.0.smithi165.stdout:5: finishing write tid 1 to smithi16530038-5 2024-09-16T04:41:44.133 INFO:tasks.rados.rados.0.smithi165.stdout:14: finishing write tid 1 to smithi16530038-14 2024-09-16T04:41:44.133 INFO:tasks.rados.rados.0.smithi165.stdout:5: finishing write tid 2 to smithi16530038-5 2024-09-16T04:41:44.136 INFO:tasks.rados.rados.0.smithi165.stdout:4: finishing write tid 3 to smithi16530038-4 2024-09-16T04:41:44.136 INFO:tasks.rados.rados.0.smithi165.stdout:4: oid 4 updating version 0 to 1 2024-09-16T04:41:44.137 INFO:tasks.rados.rados.0.smithi165.stdout:4: oid 4 updating version 1 to 2 2024-09-16T04:41:44.137 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T04:41:44.137 INFO:tasks.rados.rados.0.smithi165.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-16T04:41:44.137 INFO:tasks.rados.rados.0.smithi165.stdout:4: done (15 left) 2024-09-16T04:41:44.137 INFO:tasks.rados.rados.0.smithi165.stdout:17: write initial oid 17 2024-09-16T04:41:44.137 INFO:tasks.rados.rados.0.smithi165.stdout:17: seq_num 16 ranges {0=278528} 2024-09-16T04:41:44.140 INFO:tasks.rados.rados.0.smithi165.stdout:17: writing smithi16530038-17 from 0 to 278528 tid 1 2024-09-16T04:41:44.140 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.163 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:44.163 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:44.169 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:44.169+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:44.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:41:44.229 INFO:tasks.rados.rados.0.smithi165.stdout:8: finishing write tid 2 to smithi16530038-8 2024-09-16T04:41:44.230 INFO:tasks.rados.rados.0.smithi165.stdout:13: finishing write tid 1 to smithi16530038-13 2024-09-16T04:41:44.230 INFO:tasks.rados.rados.0.smithi165.stdout:2: finishing write tid 1 to smithi16530038-2 2024-09-16T04:41:44.231 INFO:tasks.rados.rados.0.smithi165.stdout:13: finishing write tid 2 to smithi16530038-13 2024-09-16T04:41:44.234 INFO:tasks.rados.rados.0.smithi165.stdout:14: finishing write tid 2 to smithi16530038-14 2024-09-16T04:41:44.270 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:44.268+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:44.300 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:44.300 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:44.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:41:44.317 INFO:tasks.rados.rados.0.smithi165.stdout:1: finishing write tid 1 to smithi16530038-1 2024-09-16T04:41:44.317 INFO:tasks.rados.rados.0.smithi165.stdout:1: finishing write tid 2 to smithi16530038-1 2024-09-16T04:41:44.320 INFO:tasks.rados.rados.0.smithi165.stdout:9: finishing write tid 1 to smithi16530038-9 2024-09-16T04:41:44.337 INFO:tasks.rados.rados.0.smithi165.stdout:6: finishing write tid 3 to smithi16530038-6 2024-09-16T04:41:44.337 INFO:tasks.rados.rados.0.smithi165.stdout:6: oid 6 updating version 0 to 1 2024-09-16T04:41:44.337 INFO:tasks.rados.rados.0.smithi165.stdout:6: oid 6 updating version 1 to 2 2024-09-16T04:41:44.337 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T04:41:44.337 INFO:tasks.rados.rados.0.smithi165.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:41:44.338 INFO:tasks.rados.rados.0.smithi165.stdout:6: done (15 left) 2024-09-16T04:41:44.338 INFO:tasks.rados.rados.0.smithi165.stdout:18: write initial oid 18 2024-09-16T04:41:44.338 INFO:tasks.rados.rados.0.smithi165.stdout:18: seq_num 17 ranges {0=294912} 2024-09-16T04:41:44.338 INFO:tasks.rados.rados.0.smithi165.stdout:18: writing smithi16530038-18 from 0 to 294912 tid 1 2024-09-16T04:41:44.338 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.370 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:44.368+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:44.418 INFO:tasks.rados.rados.0.smithi165.stdout:12: finishing write tid 1 to smithi16530038-12 2024-09-16T04:41:44.418 INFO:tasks.rados.rados.0.smithi165.stdout:12: finishing write tid 2 to smithi16530038-12 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:3: finishing write tid 3 to smithi16530038-3 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:3: oid 3 updating version 0 to 1 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:3: oid 3 updating version 1 to 2 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:16: finishing write tid 1 to smithi16530038-16 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:3: done (15 left) 2024-09-16T04:41:44.419 INFO:tasks.rados.rados.0.smithi165.stdout:19: write initial oid 19 2024-09-16T04:41:44.420 INFO:tasks.rados.rados.0.smithi165.stdout:19: seq_num 18 ranges {0=311296} 2024-09-16T04:41:44.421 INFO:tasks.rados.rados.0.smithi165.stdout:19: writing smithi16530038-19 from 0 to 311296 tid 1 2024-09-16T04:41:44.421 INFO:tasks.rados.rados.0.smithi165.stdout:16: finishing write tid 2 to smithi16530038-16 2024-09-16T04:41:44.422 INFO:tasks.rados.rados.0.smithi165.stdout:9: finishing write tid 2 to smithi16530038-9 2024-09-16T04:41:44.422 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.422 INFO:tasks.rados.rados.0.smithi165.stdout:15: finishing write tid 1 to smithi16530038-15 2024-09-16T04:41:44.422 INFO:tasks.rados.rados.0.smithi165.stdout:10: finishing write tid 1 to smithi16530038-10 2024-09-16T04:41:44.422 INFO:tasks.rados.rados.0.smithi165.stdout:10: finishing write tid 2 to smithi16530038-10 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:7: finishing write tid 3 to smithi16530038-7 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:7: oid 7 updating version 0 to 3 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:7: oid 7 updating version 3 to 4 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:7: done (15 left) 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:20: write initial oid 20 2024-09-16T04:41:44.435 INFO:tasks.rados.rados.0.smithi165.stdout:20: seq_num 19 ranges {0=327680} 2024-09-16T04:41:44.440 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:44.440 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:44.440 INFO:tasks.rados.rados.0.smithi165.stdout:20: writing smithi16530038-20 from 0 to 327680 tid 1 2024-09-16T04:41:44.440 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:41:44.471 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:44.468+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:44.523 INFO:tasks.rados.rados.0.smithi165.stdout:12: finishing write tid 3 to smithi16530038-12 2024-09-16T04:41:44.524 INFO:tasks.rados.rados.0.smithi165.stdout:12: oid 12 updating version 0 to 2 2024-09-16T04:41:44.524 INFO:tasks.rados.rados.0.smithi165.stdout:12: oid 12 version 2 is already newer than 1 2024-09-16T04:41:44.524 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T04:41:44.524 INFO:tasks.rados.rados.0.smithi165.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:41:44.524 INFO:tasks.rados.rados.0.smithi165.stdout:12: done (15 left) 2024-09-16T04:41:44.524 INFO:tasks.rados.rados.0.smithi165.stdout:21: write initial oid 21 2024-09-16T04:41:44.524 INFO:tasks.rados.rados.0.smithi165.stdout:21: seq_num 20 ranges {0=344064} 2024-09-16T04:41:44.527 INFO:tasks.rados.rados.0.smithi165.stdout:21: writing smithi16530038-21 from 0 to 344064 tid 1 2024-09-16T04:41:44.527 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.527 INFO:tasks.rados.rados.0.smithi165.stdout:15: finishing write tid 2 to smithi16530038-15 2024-09-16T04:41:44.535 INFO:tasks.rados.rados.0.smithi165.stdout:5: finishing write tid 3 to smithi16530038-5 2024-09-16T04:41:44.535 INFO:tasks.rados.rados.0.smithi165.stdout:5: oid 5 updating version 0 to 1 2024-09-16T04:41:44.535 INFO:tasks.rados.rados.0.smithi165.stdout:5: oid 5 updating version 1 to 2 2024-09-16T04:41:44.536 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:41:44.536 INFO:tasks.rados.rados.0.smithi165.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:41:44.536 INFO:tasks.rados.rados.0.smithi165.stdout:5: done (15 left) 2024-09-16T04:41:44.536 INFO:tasks.rados.rados.0.smithi165.stdout:22: write initial oid 22 2024-09-16T04:41:44.536 INFO:tasks.rados.rados.0.smithi165.stdout:22: seq_num 21 ranges {0=360448} 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:22: writing smithi16530038-22 from 0 to 360448 tid 1 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:14: finishing write tid 3 to smithi16530038-14 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:14: oid 14 updating version 0 to 1 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:14: oid 14 updating version 1 to 2 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:41:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:14: done (15 left) 2024-09-16T04:41:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:23: write initial oid 23 2024-09-16T04:41:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:23: seq_num 22 ranges {0=376832} 2024-09-16T04:41:44.542 INFO:tasks.rados.rados.0.smithi165.stdout:23: writing smithi16530038-23 from 0 to 376832 tid 1 2024-09-16T04:41:44.542 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.572 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:44.568+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:44.626 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:44.626 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:44.626 INFO:tasks.rados.rados.0.smithi165.stdout:9: finishing write tid 3 to smithi16530038-9 2024-09-16T04:41:44.626 INFO:tasks.rados.rados.0.smithi165.stdout:9: oid 9 updating version 0 to 1 2024-09-16T04:41:44.626 INFO:tasks.rados.rados.0.smithi165.stdout:9: oid 9 updating version 1 to 2 2024-09-16T04:41:44.627 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T04:41:44.627 INFO:tasks.rados.rados.0.smithi165.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:41:44.627 INFO:tasks.rados.rados.0.smithi165.stdout:9: done (15 left) 2024-09-16T04:41:44.627 INFO:tasks.rados.rados.0.smithi165.stdout:24: write initial oid 24 2024-09-16T04:41:44.627 INFO:tasks.rados.rados.0.smithi165.stdout:24: seq_num 23 ranges {0=393216} 2024-09-16T04:41:44.629 INFO:tasks.rados.rados.0.smithi165.stdout:24: writing smithi16530038-24 from 0 to 393216 tid 1 2024-09-16T04:41:44.629 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.629 INFO:tasks.rados.rados.0.smithi165.stdout:10: finishing write tid 3 to smithi16530038-10 2024-09-16T04:41:44.629 INFO:tasks.rados.rados.0.smithi165.stdout:10: oid 10 updating version 0 to 4 2024-09-16T04:41:44.629 INFO:tasks.rados.rados.0.smithi165.stdout:10: oid 10 version 4 is already newer than 3 2024-09-16T04:41:44.629 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 4 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T04:41:44.629 INFO:tasks.rados.rados.0.smithi165.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-16T04:41:44.630 INFO:tasks.rados.rados.0.smithi165.stdout:10: done (15 left) 2024-09-16T04:41:44.631 INFO:tasks.rados.rados.0.smithi165.stdout:25: write initial oid 25 2024-09-16T04:41:44.631 INFO:tasks.rados.rados.0.smithi165.stdout:25: seq_num 24 ranges {0=409600} 2024-09-16T04:41:44.635 INFO:tasks.rados.rados.0.smithi165.stdout:25: writing smithi16530038-25 from 0 to 409600 tid 1 2024-09-16T04:41:44.635 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.635 INFO:tasks.rados.rados.0.smithi165.stdout:19: finishing write tid 1 to smithi16530038-19 2024-09-16T04:41:44.635 INFO:tasks.rados.rados.0.smithi165.stdout:19: finishing write tid 2 to smithi16530038-19 2024-09-16T04:41:44.641 INFO:tasks.rados.rados.0.smithi165.stdout:8: finishing write tid 3 to smithi16530038-8 2024-09-16T04:41:44.641 INFO:tasks.rados.rados.0.smithi165.stdout:8: oid 8 updating version 0 to 1 2024-09-16T04:41:44.641 INFO:tasks.rados.rados.0.smithi165.stdout:8: oid 8 updating version 1 to 2 2024-09-16T04:41:44.641 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T04:41:44.641 INFO:tasks.rados.rados.0.smithi165.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:41:44.641 INFO:tasks.rados.rados.0.smithi165.stdout:1: finishing write tid 3 to smithi16530038-1 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:1: oid 1 updating version 0 to 2 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:1: oid 1 version 2 is already newer than 1 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:16: finishing write tid 3 to smithi16530038-16 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:16: oid 16 updating version 0 to 3 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:16: oid 16 updating version 3 to 4 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:1: done (15 left) 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:8: done (14 left) 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:16: done (13 left) 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:26: write initial oid 26 2024-09-16T04:41:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:26: seq_num 25 ranges {0=425984} 2024-09-16T04:41:44.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:41:44.644 INFO:tasks.rados.rados.0.smithi165.stdout:26: writing smithi16530038-26 from 0 to 425984 tid 1 2024-09-16T04:41:44.644 INFO:tasks.rados.rados.0.smithi165.stdout:27: write initial oid 27 2024-09-16T04:41:44.644 INFO:tasks.rados.rados.0.smithi165.stdout:27: seq_num 26 ranges {0=442368} 2024-09-16T04:41:44.646 INFO:tasks.rados.rados.0.smithi165.stdout:27: writing smithi16530038-27 from 0 to 442368 tid 1 2024-09-16T04:41:44.646 INFO:tasks.rados.rados.0.smithi165.stdout:28: write initial oid 28 2024-09-16T04:41:44.647 INFO:tasks.rados.rados.0.smithi165.stdout:28: seq_num 27 ranges {0=458752} 2024-09-16T04:41:44.649 INFO:tasks.rados.rados.0.smithi165.stdout:28: writing smithi16530038-28 from 0 to 458752 tid 1 2024-09-16T04:41:44.649 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:13: finishing write tid 3 to smithi16530038-13 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:13: oid 13 updating version 0 to 3 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:13: oid 13 updating version 3 to 4 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:13: done (15 left) 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:29: write initial oid 29 2024-09-16T04:41:44.650 INFO:tasks.rados.rados.0.smithi165.stdout:29: seq_num 28 ranges {0=475136} 2024-09-16T04:41:44.652 INFO:tasks.rados.rados.0.smithi165.stdout:29: writing smithi16530038-29 from 0 to 475136 tid 1 2024-09-16T04:41:44.652 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.673 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:44.669+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:44.717 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:41:44.765 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:44.765 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:44.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:41:44.785 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:44.784+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:44.826 INFO:tasks.rados.rados.0.smithi165.stdout:15: finishing write tid 3 to smithi16530038-15 2024-09-16T04:41:44.827 INFO:tasks.rados.rados.0.smithi165.stdout:15: oid 15 updating version 0 to 3 2024-09-16T04:41:44.827 INFO:tasks.rados.rados.0.smithi165.stdout:15: oid 15 updating version 3 to 4 2024-09-16T04:41:44.827 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T04:41:44.827 INFO:tasks.rados.rados.0.smithi165.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-16T04:41:44.827 INFO:tasks.rados.rados.0.smithi165.stdout:15: done (15 left) 2024-09-16T04:41:44.828 INFO:tasks.rados.rados.0.smithi165.stdout:30: write initial oid 30 2024-09-16T04:41:44.828 INFO:tasks.rados.rados.0.smithi165.stdout:30: seq_num 29 ranges {0=491520} 2024-09-16T04:41:44.831 INFO:tasks.rados.rados.0.smithi165.stdout:30: writing smithi16530038-30 from 0 to 491520 tid 1 2024-09-16T04:41:44.831 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:19: finishing write tid 3 to smithi16530038-19 2024-09-16T04:41:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:19: oid 19 updating version 0 to 1 2024-09-16T04:41:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:19: oid 19 updating version 1 to 2 2024-09-16T04:41:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T04:41:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-16T04:41:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:19: done (15 left) 2024-09-16T04:41:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:31: write initial oid 31 2024-09-16T04:41:44.833 INFO:tasks.rados.rados.0.smithi165.stdout:31: seq_num 30 ranges {0=507904} 2024-09-16T04:41:44.835 INFO:tasks.rados.rados.0.smithi165.stdout:31: writing smithi16530038-31 from 0 to 507904 tid 1 2024-09-16T04:41:44.835 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:44.835 INFO:tasks.rados.rados.0.smithi165.stdout:2: finishing write tid 2 to smithi16530038-2 2024-09-16T04:41:44.835 INFO:tasks.rados.rados.0.smithi165.stdout:11: finishing write tid 1 to smithi16530038-11 2024-09-16T04:41:44.836 INFO:tasks.rados.rados.0.smithi165.stdout:11: finishing write tid 2 to smithi16530038-11 2024-09-16T04:41:44.840 INFO:tasks.rados.rados.0.smithi165.stdout:17: finishing write tid 1 to smithi16530038-17 2024-09-16T04:41:44.853 INFO:tasks.rados.rados.0.smithi165.stdout:22: finishing write tid 1 to smithi16530038-22 2024-09-16T04:41:44.875 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:44.872+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:44.920 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:41:44.923 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:44.923 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:44.931 INFO:tasks.rados.rados.0.smithi165.stdout:21: finishing write tid 1 to smithi16530038-21 2024-09-16T04:41:44.940 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:41:44.950 INFO:tasks.rados.rados.0.smithi165.stdout:17: finishing write tid 2 to smithi16530038-17 2024-09-16T04:41:44.955 INFO:tasks.rados.rados.0.smithi165.stdout:27: finishing write tid 1 to smithi16530038-27 2024-09-16T04:41:44.956 INFO:tasks.rados.rados.0.smithi165.stdout:27: finishing write tid 2 to smithi16530038-27 2024-09-16T04:41:44.975 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:44.973+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:45.032 INFO:tasks.rados.rados.0.smithi165.stdout:21: finishing write tid 2 to smithi16530038-21 2024-09-16T04:41:45.034 INFO:tasks.rados.rados.0.smithi165.stdout:30: finishing write tid 1 to smithi16530038-30 2024-09-16T04:41:45.050 INFO:tasks.rados.rados.0.smithi165.stdout:20: finishing write tid 1 to smithi16530038-20 2024-09-16T04:41:45.051 INFO:tasks.rados.rados.0.smithi165.stdout:26: finishing write tid 1 to smithi16530038-26 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:27: finishing write tid 3 to smithi16530038-27 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:27: oid 27 updating version 0 to 6 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:27: oid 27 version 6 is already newer than 5 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:27: done (15 left) 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:32: write initial oid 32 2024-09-16T04:41:45.058 INFO:tasks.rados.rados.0.smithi165.stdout:32: seq_num 31 ranges {0=524288} 2024-09-16T04:41:45.066 INFO:tasks.rados.rados.0.smithi165.stdout:32: writing smithi16530038-32 from 0 to 524288 tid 1 2024-09-16T04:41:45.067 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.074 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:45.076 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:45.073+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:45.095 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:45.095 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:45.105 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:41:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:41:45.106 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:45.116 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:45.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:21: finishing write tid 3 to smithi16530038-21 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:21: oid 21 updating version 0 to 2 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:21: oid 21 version 2 is already newer than 1 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 2 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:21: done (15 left) 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:33: write initial oid 33 2024-09-16T04:41:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:33: seq_num 32 ranges {0=540672} 2024-09-16T04:41:45.138 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:45.138 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:45.145 INFO:tasks.rados.rados.0.smithi165.stdout:33: writing smithi16530038-33 from 0 to 540672 tid 1 2024-09-16T04:41:45.145 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.146 INFO:tasks.rados.rados.0.smithi165.stdout:30: finishing write tid 2 to smithi16530038-30 2024-09-16T04:41:45.151 INFO:tasks.rados.rados.0.smithi165.stdout:26: finishing write tid 2 to smithi16530038-26 2024-09-16T04:41:45.177 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:45.177+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:45.238 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:45.238 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:41:45.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:41:45.239 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [] 2024-09-16T04:41:45.239 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:45.250 INFO:tasks.rados.rados.0.smithi165.stdout:20: finishing write tid 2 to smithi16530038-20 2024-09-16T04:41:45.252 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:41:45.256 INFO:tasks.rados.rados.0.smithi165.stdout:17: finishing write tid 3 to smithi16530038-17 2024-09-16T04:41:45.256 INFO:tasks.rados.rados.0.smithi165.stdout:17: oid 17 updating version 0 to 6 2024-09-16T04:41:45.256 INFO:tasks.rados.rados.0.smithi165.stdout:17: oid 17 version 6 is already newer than 5 2024-09-16T04:41:45.256 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T04:41:45.257 INFO:tasks.rados.rados.0.smithi165.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-16T04:41:45.257 INFO:tasks.rados.rados.0.smithi165.stdout:17: done (15 left) 2024-09-16T04:41:45.257 INFO:tasks.rados.rados.0.smithi165.stdout:34: write initial oid 34 2024-09-16T04:41:45.257 INFO:tasks.rados.rados.0.smithi165.stdout:34: seq_num 33 ranges {0=557056} 2024-09-16T04:41:45.265 INFO:tasks.rados.rados.0.smithi165.stdout:34: writing smithi16530038-34 from 0 to 557056 tid 1 2024-09-16T04:41:45.265 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.278 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:45.272+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:45.339 INFO:tasks.rados.rados.0.smithi165.stdout:2: finishing write tid 3 to smithi16530038-2 2024-09-16T04:41:45.339 INFO:tasks.rados.rados.0.smithi165.stdout:2: oid 2 updating version 0 to 2 2024-09-16T04:41:45.339 INFO:tasks.rados.rados.0.smithi165.stdout:2: oid 2 version 2 is already newer than 1 2024-09-16T04:41:45.340 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:41:45.340 INFO:tasks.rados.rados.0.smithi165.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:41:45.340 INFO:tasks.rados.rados.0.smithi165.stdout:2: done (15 left) 2024-09-16T04:41:45.340 INFO:tasks.rados.rados.0.smithi165.stdout:35: write initial oid 35 2024-09-16T04:41:45.340 INFO:tasks.rados.rados.0.smithi165.stdout:35: seq_num 34 ranges {0=573440} 2024-09-16T04:41:45.349 INFO:tasks.rados.rados.0.smithi165.stdout:35: writing smithi16530038-35 from 0 to 573440 tid 1 2024-09-16T04:41:45.349 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.349 INFO:tasks.rados.rados.0.smithi165.stdout:11: finishing write tid 3 to smithi16530038-11 2024-09-16T04:41:45.349 INFO:tasks.rados.rados.0.smithi165.stdout:11: oid 11 updating version 0 to 4 2024-09-16T04:41:45.349 INFO:tasks.rados.rados.0.smithi165.stdout:11: oid 11 version 4 is already newer than 3 2024-09-16T04:41:45.349 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:30: finishing write tid 3 to smithi16530038-30 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:30: oid 30 updating version 0 to 3 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:30: oid 30 updating version 3 to 4 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:22: finishing write tid 2 to smithi16530038-22 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:11: done (15 left) 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:30: done (14 left) 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:36: write initial oid 36 2024-09-16T04:41:45.350 INFO:tasks.rados.rados.0.smithi165.stdout:36: seq_num 35 ranges {0=589824} 2024-09-16T04:41:45.360 INFO:tasks.rados.rados.0.smithi165.stdout:36: writing smithi16530038-36 from 0 to 589824 tid 1 2024-09-16T04:41:45.360 INFO:tasks.rados.rados.0.smithi165.stdout:37: write initial oid 37 2024-09-16T04:41:45.361 INFO:tasks.rados.rados.0.smithi165.stdout:37: seq_num 36 ranges {0=606208} 2024-09-16T04:41:45.371 INFO:tasks.rados.rados.0.smithi165.stdout:37: writing smithi16530038-37 from 0 to 606208 tid 1 2024-09-16T04:41:45.371 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.372 INFO:tasks.rados.rados.0.smithi165.stdout:23: finishing write tid 1 to smithi16530038-23 2024-09-16T04:41:45.372 INFO:tasks.rados.rados.0.smithi165.stdout:23: finishing write tid 2 to smithi16530038-23 2024-09-16T04:41:45.372 INFO:tasks.rados.rados.0.smithi165.stdout:29: finishing write tid 1 to smithi16530038-29 2024-09-16T04:41:45.378 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:45.376+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:45.423 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:45.424 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:45.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:41:45.450 INFO:tasks.rados.rados.0.smithi165.stdout:28: finishing write tid 1 to smithi16530038-28 2024-09-16T04:41:45.450 INFO:tasks.rados.rados.0.smithi165.stdout:28: finishing write tid 2 to smithi16530038-28 2024-09-16T04:41:45.450 INFO:tasks.rados.rados.0.smithi165.stdout:24: finishing write tid 1 to smithi16530038-24 2024-09-16T04:41:45.458 INFO:tasks.rados.rados.0.smithi165.stdout:26: finishing write tid 3 to smithi16530038-26 2024-09-16T04:41:45.458 INFO:tasks.rados.rados.0.smithi165.stdout:26: oid 26 updating version 0 to 6 2024-09-16T04:41:45.458 INFO:tasks.rados.rados.0.smithi165.stdout:26: oid 26 version 6 is already newer than 5 2024-09-16T04:41:45.459 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T04:41:45.459 INFO:tasks.rados.rados.0.smithi165.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-16T04:41:45.459 INFO:tasks.rados.rados.0.smithi165.stdout:26: done (15 left) 2024-09-16T04:41:45.459 INFO:tasks.rados.rados.0.smithi165.stdout:38: write initial oid 38 2024-09-16T04:41:45.459 INFO:tasks.rados.rados.0.smithi165.stdout:38: seq_num 37 ranges {0=622592} 2024-09-16T04:41:45.469 INFO:tasks.rados.rados.0.smithi165.stdout:38: writing smithi16530038-38 from 0 to 622592 tid 1 2024-09-16T04:41:45.470 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.480 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:45.476+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:45.550 INFO:tasks.rados.rados.0.smithi165.stdout:24: finishing write tid 2 to smithi16530038-24 2024-09-16T04:41:45.553 INFO:tasks.rados.rados.0.smithi165.stdout:31: finishing write tid 1 to smithi16530038-31 2024-09-16T04:41:45.553 INFO:tasks.rados.rados.0.smithi165.stdout:31: finishing write tid 2 to smithi16530038-31 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:20: finishing write tid 3 to smithi16530038-20 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:20: oid 20 updating version 0 to 1 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:20: oid 20 updating version 1 to 2 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:20: done (15 left) 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:39: write initial oid 39 2024-09-16T04:41:45.559 INFO:tasks.rados.rados.0.smithi165.stdout:39: seq_num 38 ranges {0=638976} 2024-09-16T04:41:45.569 INFO:tasks.rados.rados.0.smithi165.stdout:39: writing smithi16530038-39 from 0 to 638976 tid 1 2024-09-16T04:41:45.569 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.570 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:45.570 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:45.572 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:45.572 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:45.581 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:45.580+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:45.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:41:45.653 INFO:tasks.rados.rados.0.smithi165.stdout:28: finishing write tid 3 to smithi16530038-28 2024-09-16T04:41:45.653 INFO:tasks.rados.rados.0.smithi165.stdout:28: oid 28 updating version 0 to 5 2024-09-16T04:41:45.654 INFO:tasks.rados.rados.0.smithi165.stdout:28: oid 28 updating version 5 to 6 2024-09-16T04:41:45.654 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T04:41:45.654 INFO:tasks.rados.rados.0.smithi165.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-16T04:41:45.654 INFO:tasks.rados.rados.0.smithi165.stdout:28: done (15 left) 2024-09-16T04:41:45.654 INFO:tasks.rados.rados.0.smithi165.stdout:40: write initial oid 40 2024-09-16T04:41:45.654 INFO:tasks.rados.rados.0.smithi165.stdout:40: seq_num 39 ranges {0=671744} 2024-09-16T04:41:45.665 INFO:tasks.rados.rados.0.smithi165.stdout:40: writing smithi16530038-40 from 0 to 671744 tid 1 2024-09-16T04:41:45.665 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:35: finishing write tid 1 to smithi16530038-35 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:35: finishing write tid 2 to smithi16530038-35 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:22: finishing write tid 3 to smithi16530038-22 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:22: oid 22 updating version 0 to 2 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:22: oid 22 version 2 is already newer than 1 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 2 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:23: finishing write tid 3 to smithi16530038-23 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:23: oid 23 updating version 0 to 4 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:23: oid 23 version 4 is already newer than 3 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 4 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:41:45.666 INFO:tasks.rados.rados.0.smithi165.stdout:22: done (15 left) 2024-09-16T04:41:45.667 INFO:tasks.rados.rados.0.smithi165.stdout:23: done (14 left) 2024-09-16T04:41:45.667 INFO:tasks.rados.rados.0.smithi165.stdout:41: write initial oid 41 2024-09-16T04:41:45.667 INFO:tasks.rados.rados.0.smithi165.stdout:41: seq_num 40 ranges {0=688128} 2024-09-16T04:41:45.677 INFO:tasks.rados.rados.0.smithi165.stdout:41: writing smithi16530038-41 from 0 to 688128 tid 1 2024-09-16T04:41:45.677 INFO:tasks.rados.rados.0.smithi165.stdout:42: write initial oid 42 2024-09-16T04:41:45.677 INFO:tasks.rados.rados.0.smithi165.stdout:42: seq_num 41 ranges {0=704512} 2024-09-16T04:41:45.681 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:45.682 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:45.681+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:45.690 INFO:tasks.rados.rados.0.smithi165.stdout:42: writing smithi16530038-42 from 0 to 704512 tid 1 2024-09-16T04:41:45.690 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.691 INFO:tasks.rados.rados.0.smithi165.stdout:36: finishing write tid 1 to smithi16530038-36 2024-09-16T04:41:45.691 INFO:tasks.rados.rados.0.smithi165.stdout:36: finishing write tid 2 to smithi16530038-36 2024-09-16T04:41:45.691 INFO:tasks.rados.rados.0.smithi165.stdout:37: finishing write tid 1 to smithi16530038-37 2024-09-16T04:41:45.691 INFO:tasks.rados.rados.0.smithi165.stdout:37: finishing write tid 2 to smithi16530038-37 2024-09-16T04:41:45.708 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:45.708 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:45.742 INFO:tasks.rados.rados.0.smithi165.stdout:18: finishing write tid 1 to smithi16530038-18 2024-09-16T04:41:45.742 INFO:tasks.rados.rados.0.smithi165.stdout:18: finishing write tid 2 to smithi16530038-18 2024-09-16T04:41:45.743 INFO:tasks.rados.rados.0.smithi165.stdout:25: finishing write tid 1 to smithi16530038-25 2024-09-16T04:41:45.743 INFO:tasks.rados.rados.0.smithi165.stdout:25: finishing write tid 2 to smithi16530038-25 2024-09-16T04:41:45.756 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:41:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:41:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [] 2024-09-16T04:41:45.756 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:24: finishing write tid 3 to smithi16530038-24 2024-09-16T04:41:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:24: oid 24 updating version 0 to 3 2024-09-16T04:41:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:24: oid 24 updating version 3 to 4 2024-09-16T04:41:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:41:45.758 INFO:tasks.rados.rados.0.smithi165.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-16T04:41:45.758 INFO:tasks.rados.rados.0.smithi165.stdout:24: done (15 left) 2024-09-16T04:41:45.758 INFO:tasks.rados.rados.0.smithi165.stdout:43: write initial oid 43 2024-09-16T04:41:45.758 INFO:tasks.rados.rados.0.smithi165.stdout:43: seq_num 42 ranges {0=720896} 2024-09-16T04:41:45.770 INFO:tasks.rados.rados.0.smithi165.stdout:43: writing smithi16530038-43 from 0 to 720896 tid 1 2024-09-16T04:41:45.770 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.773 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:41:45.782 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:45.780+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:45.809 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:45.810 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:45.858 INFO:tasks.rados.rados.0.smithi165.stdout:35: finishing write tid 3 to smithi16530038-35 2024-09-16T04:41:45.858 INFO:tasks.rados.rados.0.smithi165.stdout:35: oid 35 updating version 0 to 1 2024-09-16T04:41:45.858 INFO:tasks.rados.rados.0.smithi165.stdout:35: oid 35 updating version 1 to 2 2024-09-16T04:41:45.858 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 2 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T04:41:45.858 INFO:tasks.rados.rados.0.smithi165.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-16T04:41:45.858 INFO:tasks.rados.rados.0.smithi165.stdout:35: done (15 left) 2024-09-16T04:41:45.859 INFO:tasks.rados.rados.0.smithi165.stdout:44: write initial oid 44 2024-09-16T04:41:45.859 INFO:tasks.rados.rados.0.smithi165.stdout:44: seq_num 43 ranges {0=737280} 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:44: writing smithi16530038-44 from 0 to 737280 tid 1 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:31: finishing write tid 3 to smithi16530038-31 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:31: oid 31 updating version 0 to 5 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:31: oid 31 updating version 5 to 6 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:32: finishing write tid 1 to smithi16530038-32 2024-09-16T04:41:45.871 INFO:tasks.rados.rados.0.smithi165.stdout:32: finishing write tid 2 to smithi16530038-32 2024-09-16T04:41:45.872 INFO:tasks.rados.rados.0.smithi165.stdout:34: finishing write tid 1 to smithi16530038-34 2024-09-16T04:41:45.872 INFO:tasks.rados.rados.0.smithi165.stdout:31: done (15 left) 2024-09-16T04:41:45.872 INFO:tasks.rados.rados.0.smithi165.stdout:45: write initial oid 45 2024-09-16T04:41:45.872 INFO:tasks.rados.rados.0.smithi165.stdout:45: seq_num 44 ranges {0=753664} 2024-09-16T04:41:45.883 INFO:tasks.rados.rados.0.smithi165.stdout:45: writing smithi16530038-45 from 0 to 753664 tid 1 2024-09-16T04:41:45.883 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.883 INFO:tasks.rados.rados.0.smithi165.stdout:38: finishing write tid 1 to smithi16530038-38 2024-09-16T04:41:45.884 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:45.881+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:45.895 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:45.896 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:45.899 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:45.899 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:45.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:41:45.936 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:45.958 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:45.958 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:45.964 INFO:tasks.rados.rados.0.smithi165.stdout:29: finishing write tid 2 to smithi16530038-29 2024-09-16T04:41:45.966 INFO:tasks.rados.rados.0.smithi165.stdout:34: finishing write tid 2 to smithi16530038-34 2024-09-16T04:41:45.969 INFO:tasks.rados.rados.0.smithi165.stdout:36: finishing write tid 3 to smithi16530038-36 2024-09-16T04:41:45.969 INFO:tasks.rados.rados.0.smithi165.stdout:36: oid 36 updating version 0 to 7 2024-09-16T04:41:45.970 INFO:tasks.rados.rados.0.smithi165.stdout:36: oid 36 updating version 7 to 8 2024-09-16T04:41:45.970 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T04:41:45.970 INFO:tasks.rados.rados.0.smithi165.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-16T04:41:45.970 INFO:tasks.rados.rados.0.smithi165.stdout:36: done (15 left) 2024-09-16T04:41:45.970 INFO:tasks.rados.rados.0.smithi165.stdout:46: write initial oid 46 2024-09-16T04:41:45.970 INFO:tasks.rados.rados.0.smithi165.stdout:46: seq_num 45 ranges {0=770048} 2024-09-16T04:41:45.982 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:45.982 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:45.983 INFO:tasks.rados.rados.0.smithi165.stdout:46: writing smithi16530038-46 from 0 to 770048 tid 1 2024-09-16T04:41:45.983 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:45.983 INFO:tasks.rados.rados.0.smithi165.stdout:43: finishing write tid 1 to smithi16530038-43 2024-09-16T04:41:45.984 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:45.980+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:46.005 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:46.005 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:46.029 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:46.030 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:46.064 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:46.064 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:46.064 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:41:46.064 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:41:46.065 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:46.066 INFO:tasks.rados.rados.0.smithi165.stdout:43: finishing write tid 2 to smithi16530038-43 2024-09-16T04:41:46.068 INFO:tasks.rados.rados.0.smithi165.stdout:37: finishing write tid 3 to smithi16530038-37 2024-09-16T04:41:46.068 INFO:tasks.rados.rados.0.smithi165.stdout:37: oid 37 updating version 0 to 6 2024-09-16T04:41:46.068 INFO:tasks.rados.rados.0.smithi165.stdout:37: oid 37 version 6 is already newer than 5 2024-09-16T04:41:46.068 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:41:46.068 INFO:tasks.rados.rados.0.smithi165.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:41:46.069 INFO:tasks.rados.rados.0.smithi165.stdout:37: done (15 left) 2024-09-16T04:41:46.069 INFO:tasks.rados.rados.0.smithi165.stdout:47: write initial oid 47 2024-09-16T04:41:46.069 INFO:tasks.rados.rados.0.smithi165.stdout:47: seq_num 46 ranges {0=786432} 2024-09-16T04:41:46.080 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:41:46.082 INFO:tasks.rados.rados.0.smithi165.stdout:47: writing smithi16530038-47 from 0 to 786432 tid 1 2024-09-16T04:41:46.082 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:46.082 INFO:tasks.rados.rados.0.smithi165.stdout:41: finishing write tid 1 to smithi16530038-41 2024-09-16T04:41:46.082 INFO:tasks.rados.rados.0.smithi165.stdout:38: finishing write tid 2 to smithi16530038-38 2024-09-16T04:41:46.085 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:46.085+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:46.104 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:46.104 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:46.162 INFO:tasks.rados.rados.0.smithi165.stdout:39: finishing write tid 1 to smithi16530038-39 2024-09-16T04:41:46.162 INFO:tasks.rados.rados.0.smithi165.stdout:39: finishing write tid 2 to smithi16530038-39 2024-09-16T04:41:46.168 INFO:tasks.rados.rados.0.smithi165.stdout:29: finishing write tid 3 to smithi16530038-29 2024-09-16T04:41:46.168 INFO:tasks.rados.rados.0.smithi165.stdout:29: oid 29 updating version 0 to 5 2024-09-16T04:41:46.168 INFO:tasks.rados.rados.0.smithi165.stdout:29: oid 29 updating version 5 to 6 2024-09-16T04:41:46.168 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T04:41:46.168 INFO:tasks.rados.rados.0.smithi165.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-16T04:41:46.168 INFO:tasks.rados.rados.0.smithi165.stdout:29: done (15 left) 2024-09-16T04:41:46.169 INFO:tasks.rados.rados.0.smithi165.stdout:48: write initial oid 48 2024-09-16T04:41:46.169 INFO:tasks.rados.rados.0.smithi165.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-16T04:41:46.175 INFO:tasks.rados.rados.0.smithi165.stdout:48: writing smithi16530038-48 from 0 to 425984 tid 1 2024-09-16T04:41:46.182 INFO:tasks.rados.rados.0.smithi165.stdout:48: writing smithi16530038-48 from 425984 to 802816 tid 2 2024-09-16T04:41:46.182 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:46.182 INFO:tasks.rados.rados.0.smithi165.stdout:32: finishing write tid 3 to smithi16530038-32 2024-09-16T04:41:46.182 INFO:tasks.rados.rados.0.smithi165.stdout:32: oid 32 updating version 0 to 8 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:32: oid 32 version 8 is already newer than 7 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 8 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:34: finishing write tid 3 to smithi16530038-34 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:34: oid 34 updating version 0 to 9 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:34: oid 34 updating version 9 to 10 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 10 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:43: finishing write tid 3 to smithi16530038-43 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:43: oid 43 updating version 0 to 4 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:43: oid 43 version 4 is already newer than 3 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 4 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:41:46.183 INFO:tasks.rados.rados.0.smithi165.stdout:44: finishing write tid 1 to smithi16530038-44 2024-09-16T04:41:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:40: finishing write tid 1 to smithi16530038-40 2024-09-16T04:41:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:40: finishing write tid 2 to smithi16530038-40 2024-09-16T04:41:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:32: done (15 left) 2024-09-16T04:41:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:34: done (14 left) 2024-09-16T04:41:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:43: done (13 left) 2024-09-16T04:41:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:49: write initial oid 49 2024-09-16T04:41:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-16T04:41:46.186 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:46.180+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:46.190 INFO:tasks.rados.rados.0.smithi165.stdout:49: writing smithi16530038-49 from 0 to 442368 tid 1 2024-09-16T04:41:46.196 INFO:tasks.rados.rados.0.smithi165.stdout:49: writing smithi16530038-49 from 442368 to 819200 tid 2 2024-09-16T04:41:46.196 INFO:tasks.rados.rados.0.smithi165.stdout:50: write initial oid 50 2024-09-16T04:41:46.197 INFO:tasks.rados.rados.0.smithi165.stdout:41: finishing write tid 2 to smithi16530038-41 2024-09-16T04:41:46.197 INFO:tasks.rados.rados.0.smithi165.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-16T04:41:46.204 INFO:tasks.rados.rados.0.smithi165.stdout:50: writing smithi16530038-50 from 0 to 458752 tid 1 2024-09-16T04:41:46.210 INFO:tasks.rados.rados.0.smithi165.stdout:50: writing smithi16530038-50 from 458752 to 835584 tid 2 2024-09-16T04:41:46.210 INFO:tasks.rados.rados.0.smithi165.stdout:51: copy_from oid 20 from oid 11 current snap is 0 2024-09-16T04:41:46.211 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:46.215 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:46.216 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/32 ECSubWrite(tid=16, reqid=client.4728.0:124, at_version=36'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.945658+0000", 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.283991938, 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015200135, 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.230 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.231 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.231 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.231 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945658+0000", 2024-09-16T04:41:46.231 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945658+0000", 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945659+0000", 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.765e-06 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945786+0000", 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126704 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945804+0000", 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8275999999999999e-05 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945819+0000", 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4229999999999999e-05 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.959704+0000", 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013885378 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.959710+0000", 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7459999999999997e-06 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.960819+0000", 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001109089 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.960858+0000", 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8946999999999998e-05 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.235 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/32 ECSubWrite(tid=17, reqid=client.4728.0:125, at_version=36'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.945836+0000", 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.28381313400000002, 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015069872, 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945836+0000", 2024-09-16T04:41:46.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945836+0000", 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945837+0000", 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2300000000000001e-07 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945856+0000", 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9086e-05 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945872+0000", 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5491000000000001e-05 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945882+0000", 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0645000000000001e-05 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.960162+0000", 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014280085 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.960171+0000", 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1069999999999994e-06 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:46.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.960871+0000", 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00069952100000000004 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.960906+0000", 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5413999999999997e-05 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.240 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 36/32 ECSubWriteReply(tid=10, last_complete=36'6, committed=1, applied=1))", 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.945962+0000", 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.28368784000000002, 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.122220231, 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945962+0000", 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.241 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945962+0000", 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945963+0000", 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.186e-06 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945970+0000", 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2830000000000004e-06 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.945991+0000", 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0446e-05 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.046105+0000", 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100114446 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.243 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068120+0000", 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.022014946000000001 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068182+0000", 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1923999999999999e-05 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.244 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.cs1 36/32 ECSubRead(tid=24, to_read={3:372a9655:::smithi16530038-32:head=0,4096,0}, subchunks={3:372a9655:::smithi16530038-32:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.967390+0000", 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.26225957900000002, 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10067656799999999, 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967390+0000", 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.245 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967390+0000", 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967391+0000", 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.389e-06 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967405+0000", 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3634e-05 2024-09-16T04:41:46.246 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967429+0000", 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3929000000000001e-05 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967444+0000", 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5126000000000001e-05 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.067686+0000", 2024-09-16T04:41:46.247 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100241885 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068067+0000", 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00038060500000000001 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs4 36/32 ECSubWrite(tid=25, reqid=client.4728.0:86, at_version=0'0, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.967481+0000", 2024-09-16T04:41:46.248 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.26216827999999998, 2024-09-16T04:41:46.249 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011102550000000001, 2024-09-16T04:41:46.249 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.249 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.249 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.249 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.249 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967481+0000", 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967481+0000", 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967482+0000", 2024-09-16T04:41:46.250 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0200000000000002e-07 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967495+0000", 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2721999999999999e-05 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967510+0000", 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5917e-05 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.251 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967520+0000", 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9480000000000003e-06 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967585+0000", 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4603999999999993e-05 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967594+0000", 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6989999999999992e-06 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.252 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968541+0000", 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00094709600000000003 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968592+0000", 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0766999999999997e-05 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.253 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 36/32 ECSubWrite(tid=26, reqid=client.4728.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.967678+0000", 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.261971485, 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101411099, 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967678+0000", 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.254 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967678+0000", 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967679+0000", 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0500000000000002e-07 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967690+0000", 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1129e-05 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.255 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967705+0000", 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4755e-05 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.967717+0000", 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1765e-05 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068168+0000", 2024-09-16T04:41:46.256 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10045193099999999 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068182+0000", 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3263e-05 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.069011+0000", 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00082890100000000005 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.069089+0000", 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8549999999999998e-05 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.cs1 36/32 ECSubRead(tid=27, to_read={3:376489e5:::smithi16530038-34:head=0,4096,0}, subchunks={3:376489e5:::smithi16530038-34:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.968720+0000", 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.26092988, 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100278098, 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.258 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968720+0000", 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968720+0000", 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968721+0000", 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9600000000000008e-07 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968731+0000", 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0356999999999999e-05 2024-09-16T04:41:46.260 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968752+0000", 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0965e-05 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.968770+0000", 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7876e-05 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068605+0000", 2024-09-16T04:41:46.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.099835483000000003 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068998+0000", 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000392421 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.4s0 36/32 ECSubReadReply(tid=11, attrs_read=0))", 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.970355+0000", 2024-09-16T04:41:46.262 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.25929485000000002, 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098532518999999999, 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.970355+0000", 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.970355+0000", 2024-09-16T04:41:46.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.970356+0000", 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2999999999999999e-07 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.970371+0000", 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.508e-05 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.264 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.970387+0000", 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6455000000000001e-05 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.970398+0000", 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0373999999999999e-05 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068574+0000", 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098176158999999999 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.265 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.068887+0000", 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031352099999999998 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.9s5 36/32 ECSubWrite(tid=17, reqid=client.4728.0:139, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.087015+0000", 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.142634182, 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001846856, 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.266 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087015+0000", 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087015+0000", 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.267 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087017+0000", 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.488e-06 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087648+0000", 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063111700000000005 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087683+0000", 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5178000000000001e-05 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.268 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087701+0000", 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7450000000000001e-05 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087814+0000", 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000113305 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.087825+0000", 2024-09-16T04:41:46.269 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0933e-05 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.088802+0000", 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00097759600000000006 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.088862+0000", 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9789e-05 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.270 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.271 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.271 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.271 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:46.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:41:46.298 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 1 to smithi16530038-49 2024-09-16T04:41:46.317 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:46.316+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:46.364 INFO:tasks.rados.rados.0.smithi165.stdout:47: finishing write tid 1 to smithi16530038-47 2024-09-16T04:41:46.367 INFO:tasks.rados.rados.0.smithi165.stdout:39: finishing write tid 3 to smithi16530038-39 2024-09-16T04:41:46.367 INFO:tasks.rados.rados.0.smithi165.stdout:39: oid 39 updating version 0 to 8 2024-09-16T04:41:46.367 INFO:tasks.rados.rados.0.smithi165.stdout:39: oid 39 version 8 is already newer than 7 2024-09-16T04:41:46.367 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 8 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T04:41:46.368 INFO:tasks.rados.rados.0.smithi165.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-16T04:41:46.368 INFO:tasks.rados.rados.0.smithi165.stdout:39: done (15 left) 2024-09-16T04:41:46.368 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 27 current snap is 0 2024-09-16T04:41:46.368 INFO:tasks.rados.rados.0.smithi165.stdout:52: seq_num 50 ranges {442368=475136,917504=376832} 2024-09-16T04:41:46.382 INFO:tasks.rados.rados.0.smithi165.stdout:52: writing smithi16530038-27 from 442368 to 917504 tid 1 2024-09-16T04:41:46.388 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:46.384+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:46.389 INFO:tasks.rados.rados.0.smithi165.stdout:52: writing smithi16530038-27 from 917504 to 1294336 tid 2 2024-09-16T04:41:46.389 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:46.389 INFO:tasks.rados.rados.0.smithi165.stdout:46: finishing write tid 1 to smithi16530038-46 2024-09-16T04:41:46.389 INFO:tasks.rados.rados.0.smithi165.stdout:44: finishing write tid 2 to smithi16530038-44 2024-09-16T04:41:46.391 INFO:tasks.rados.rados.0.smithi165.stdout:45: finishing write tid 1 to smithi16530038-45 2024-09-16T04:41:46.391 INFO:tasks.rados.rados.0.smithi165.stdout:45: finishing write tid 2 to smithi16530038-45 2024-09-16T04:41:46.393 INFO:tasks.rados.rados.0.smithi165.stdout:33: finishing write tid 1 to smithi16530038-33 2024-09-16T04:41:46.401 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:46.401 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:46.401 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:41:46.401 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:46.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:41:46.453 INFO:tasks.rados.rados.0.smithi165.stdout:18: finishing write tid 3 to smithi16530038-18 2024-09-16T04:41:46.453 INFO:tasks.rados.rados.0.smithi165.stdout:18: oid 18 updating version 0 to 6 2024-09-16T04:41:46.453 INFO:tasks.rados.rados.0.smithi165.stdout:18: oid 18 version 6 is already newer than 5 2024-09-16T04:41:46.453 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T04:41:46.453 INFO:tasks.rados.rados.0.smithi165.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-16T04:41:46.453 INFO:tasks.rados.rados.0.smithi165.stdout:18: done (15 left) 2024-09-16T04:41:46.453 INFO:tasks.rados.rados.0.smithi165.stdout:53: rmattr oid 31 current snap is 0 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:53: done (15 left) 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:54: delete oid 30 current snap is 0 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:25: finishing write tid 3 to smithi16530038-25 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:25: oid 25 updating version 0 to 7 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:25: oid 25 updating version 7 to 8 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 8 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:33: finishing write tid 2 to smithi16530038-33 2024-09-16T04:41:46.454 INFO:tasks.rados.rados.0.smithi165.stdout:42: finishing write tid 1 to smithi16530038-42 2024-09-16T04:41:46.489 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:46.489+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:46.497 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 1 to smithi16530038-50 2024-09-16T04:41:46.498 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 2 to smithi16530038-49 2024-09-16T04:41:46.555 INFO:tasks.rados.rados.0.smithi165.stdout:42: finishing write tid 2 to smithi16530038-42 2024-09-16T04:41:46.568 INFO:tasks.rados.rados.0.smithi165.stdout:46: finishing write tid 2 to smithi16530038-46 2024-09-16T04:41:46.585 INFO:tasks.rados.rados.0.smithi165.stdout:44: finishing write tid 3 to smithi16530038-44 2024-09-16T04:41:46.585 INFO:tasks.rados.rados.0.smithi165.stdout:44: oid 44 updating version 0 to 4 2024-09-16T04:41:46.585 INFO:tasks.rados.rados.0.smithi165.stdout:44: oid 44 version 4 is already newer than 3 2024-09-16T04:41:46.585 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T04:41:46.585 INFO:tasks.rados.rados.0.smithi165.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-16T04:41:46.586 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:46.586 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:46.586 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:41:46.586 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:41:46.586 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:46.587 INFO:tasks.rados.rados.0.smithi165.stdout:38: finishing write tid 3 to smithi16530038-38 2024-09-16T04:41:46.587 INFO:tasks.rados.rados.0.smithi165.stdout:38: oid 38 updating version 0 to 4 2024-09-16T04:41:46.587 INFO:tasks.rados.rados.0.smithi165.stdout:38: oid 38 version 4 is already newer than 3 2024-09-16T04:41:46.587 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:41:46.587 INFO:tasks.rados.rados.0.smithi165.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:41:46.588 INFO:tasks.rados.rados.0.smithi165.stdout:47: finishing write tid 2 to smithi16530038-47 2024-09-16T04:41:46.589 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:46.584+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:46.591 INFO:tasks.rados.rados.0.smithi165.stdout:41: finishing write tid 3 to smithi16530038-41 2024-09-16T04:41:46.591 INFO:tasks.rados.rados.0.smithi165.stdout:41: oid 41 updating version 0 to 6 2024-09-16T04:41:46.591 INFO:tasks.rados.rados.0.smithi165.stdout:41: oid 41 version 6 is already newer than 5 2024-09-16T04:41:46.591 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:41:46.591 INFO:tasks.rados.rados.0.smithi165.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:41:46.598 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 2 to smithi16530038-50 2024-09-16T04:41:46.599 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 3 to smithi16530038-50 2024-09-16T04:41:46.600 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:41:46.658 INFO:tasks.rados.rados.0.smithi165.stdout:40: finishing write tid 3 to smithi16530038-40 2024-09-16T04:41:46.658 INFO:tasks.rados.rados.0.smithi165.stdout:40: oid 40 updating version 0 to 5 2024-09-16T04:41:46.658 INFO:tasks.rados.rados.0.smithi165.stdout:40: oid 40 updating version 5 to 6 2024-09-16T04:41:46.658 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T04:41:46.658 INFO:tasks.rados.rados.0.smithi165.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-16T04:41:46.659 INFO:tasks.rados.rados.0.smithi165.stdout:33: finishing write tid 3 to smithi16530038-33 2024-09-16T04:41:46.659 INFO:tasks.rados.rados.0.smithi165.stdout:33: oid 33 updating version 0 to 10 2024-09-16T04:41:46.659 INFO:tasks.rados.rados.0.smithi165.stdout:33: oid 33 version 10 is already newer than 9 2024-09-16T04:41:46.659 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 10 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T04:41:46.659 INFO:tasks.rados.rados.0.smithi165.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:41:46.690 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:46.688+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_repop(client.4356.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.222551843000002, 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021469340000000001, 2024-09-16T04:41:46.728 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505677+0000", 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3090000000000001e-06 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505703+0000", 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6139000000000002e-05 2024-09-16T04:41:46.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505720+0000", 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7329e-05 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505886+0000", 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016637499999999999 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505945+0000", 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8699999999999997e-05 2024-09-16T04:41:46.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.507713+0000", 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001768218 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.507814+0000", 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100274 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_repop(client.4356.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:29.620732+0000", 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.107486050999999, 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00184699, 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620732+0000", 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620732+0000", 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620734+0000", 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8199999999999999e-06 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620747+0000", 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2323e-05 2024-09-16T04:41:46.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620775+0000", 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8824999999999999e-05 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620791+0000", 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6056e-05 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620970+0000", 2024-09-16T04:41:46.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017873800000000001 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.621012+0000", 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1653999999999998e-05 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.622489+0000", 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014767059999999999 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:46.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.622579+0000", 2024-09-16T04:41:46.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0867999999999998e-05 2024-09-16T04:41:46.737 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.737 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.737 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.737 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:46.737 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:46.737 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:46.741 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:41:46.743 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:41:46.788 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 3 to smithi16530038-49 2024-09-16T04:41:46.790 INFO:tasks.rados.rados.0.smithi165.stdout:52: finishing write tid 1 to smithi16530038-27 2024-09-16T04:41:46.791 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:46.789+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:46.801 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 4 to smithi16530038-50 2024-09-16T04:41:46.802 INFO:tasks.rados.rados.0.smithi165.stdout:50: oid 50 updating version 0 to 1 2024-09-16T04:41:46.802 INFO:tasks.rados.rados.0.smithi165.stdout:50: oid 50 updating version 1 to 3 2024-09-16T04:41:46.802 INFO:tasks.rados.rados.0.smithi165.stdout:50: oid 50 version 3 is already newer than 2 2024-09-16T04:41:46.802 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 3 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T04:41:46.802 INFO:tasks.rados.rados.0.smithi165.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-16T04:41:46.872 INFO:tasks.rados.rados.0.smithi165.stdout:46: finishing write tid 3 to smithi16530038-46 2024-09-16T04:41:46.872 INFO:tasks.rados.rados.0.smithi165.stdout:46: oid 46 updating version 0 to 10 2024-09-16T04:41:46.872 INFO:tasks.rados.rados.0.smithi165.stdout:46: oid 46 version 10 is already newer than 9 2024-09-16T04:41:46.872 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 10 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T04:41:46.872 INFO:tasks.rados.rados.0.smithi165.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-16T04:41:46.874 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:46.874 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:46.875 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:41:46.875 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:46.889 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:41:46.892 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:46.888+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:46.892 INFO:tasks.rados.rados.0.smithi165.stdout:45: finishing write tid 3 to smithi16530038-45 2024-09-16T04:41:46.892 INFO:tasks.rados.rados.0.smithi165.stdout:45: oid 45 updating version 0 to 8 2024-09-16T04:41:46.892 INFO:tasks.rados.rados.0.smithi165.stdout:45: oid 45 version 8 is already newer than 7 2024-09-16T04:41:46.893 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T04:41:46.893 INFO:tasks.rados.rados.0.smithi165.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-16T04:41:46.896 INFO:tasks.rados.rados.0.smithi165.stdout:25: done (15 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:33: done (14 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:38: done (13 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:40: done (12 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:41: done (11 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:44: done (10 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:45: done (9 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:46: done (8 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:50: done (7 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:54: done (6 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:55: read oid 32 snap -1 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:55: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:56: rollback oid 39 current snap is 0 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:56: done (7 left) 2024-09-16T04:41:46.897 INFO:tasks.rados.rados.0.smithi165.stdout:57: read oid 5 snap -1 2024-09-16T04:41:46.898 INFO:tasks.rados.rados.0.smithi165.stdout:57: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:41:46.898 INFO:tasks.rados.rados.0.smithi165.stdout:58: copy_from oid 30 from oid 31 current snap is 0 2024-09-16T04:41:46.898 INFO:tasks.rados.rados.0.smithi165.stdout:59: rollback oid 15 current snap is 0 2024-09-16T04:41:46.898 INFO:tasks.rados.rados.0.smithi165.stdout:59: done (9 left) 2024-09-16T04:41:46.898 INFO:tasks.rados.rados.0.smithi165.stdout:60: copy_from oid 26 from oid 2 current snap is 0 2024-09-16T04:41:46.898 INFO:tasks.rados.rados.0.smithi165.stdout:61: delete oid 41 current snap is 0 2024-09-16T04:41:46.899 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 4 to smithi16530038-49 2024-09-16T04:41:46.899 INFO:tasks.rados.rados.0.smithi165.stdout:49: oid 49 updating version 0 to 8 2024-09-16T04:41:46.899 INFO:tasks.rados.rados.0.smithi165.stdout:49: oid 49 version 8 is already newer than 7 2024-09-16T04:41:46.899 INFO:tasks.rados.rados.0.smithi165.stdout:49: oid 49 updating version 8 to 9 2024-09-16T04:41:46.899 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:41:46.899 INFO:tasks.rados.rados.0.smithi165.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-16T04:41:46.993 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:46.992+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:46.995 INFO:tasks.rados.rados.0.smithi165.stdout:47: finishing write tid 3 to smithi16530038-47 2024-09-16T04:41:46.996 INFO:tasks.rados.rados.0.smithi165.stdout:47: oid 47 updating version 0 to 8 2024-09-16T04:41:46.996 INFO:tasks.rados.rados.0.smithi165.stdout:47: oid 47 version 8 is already newer than 7 2024-09-16T04:41:46.996 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T04:41:46.996 INFO:tasks.rados.rados.0.smithi165.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-16T04:41:46.996 INFO:tasks.rados.rados.0.smithi165.stdout:52: finishing write tid 2 to smithi16530038-27 2024-09-16T04:41:47.001 INFO:tasks.rados.rados.0.smithi165.stdout:52: finishing write tid 3 to smithi16530038-27 2024-09-16T04:41:47.010 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:47.010 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:47.010 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:41:47.010 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:41:47.011 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:47.024 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:41:47.094 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:47.092+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:47.099 INFO:tasks.rados.rados.0.smithi165.stdout:58: finishing copy_from racing read to smithi16530038-30 2024-09-16T04:41:47.158 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:47.159 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:41:47.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:41:47.159 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:41:47.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.159 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=11, reqid=client.4728.0:103, at_version=36'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:47.159 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.353387+0000", 2024-09-16T04:41:47.159 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.801719359, 2024-09-16T04:41:47.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00254383, 2024-09-16T04:41:47.160 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.353387+0000", 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.353387+0000", 2024-09-16T04:41:47.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.162 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.163 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.163 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.353389+0000", 2024-09-16T04:41:47.163 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9970000000000001e-06 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.353807+0000", 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041857 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.353847+0000", 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9113999999999999e-05 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.353868+0000", 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1342999999999999e-05 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.354029+0000", 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016080000000000001 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.354057+0000", 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8286e-05 2024-09-16T04:41:47.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.355845+0000", 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017884039999999999 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.355931+0000", 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5315999999999994e-05 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=8, reqid=client.4728.0:109, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.373789+0000", 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.7813174219999999, 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002624843, 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.373789+0000", 2024-09-16T04:41:47.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.373789+0000", 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.373791+0000", 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0430000000000002e-06 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.374116+0000", 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00032517000000000001 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.374147+0000", 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1408999999999997e-05 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.374163+0000", 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5515999999999998e-05 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.374331+0000", 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016796000000000001 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.374357+0000", 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6157999999999999e-05 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.376324+0000", 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019673630000000002 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.376414+0000", 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9223999999999998e-05 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=9, reqid=client.4728.0:121, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.681150+0000", 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.473956668, 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027773099999999998, 2024-09-16T04:41:47.171 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.681150+0000", 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.681150+0000", 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.681152+0000", 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.681815+0000", 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000663526 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.681844+0000", 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8535999999999999e-05 2024-09-16T04:41:47.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.681860+0000", 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6149000000000001e-05 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.682007+0000", 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014704400000000001 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.682034+0000", 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7446000000000001e-05 2024-09-16T04:41:47.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.683859+0000", 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001825215 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.683927+0000", 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7473999999999998e-05 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=10, reqid=client.4728.0:122, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.471056546, 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201524707, 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:41:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684051+0000", 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.277e-06 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684081+0000", 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0414999999999999e-05 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684109+0000", 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7455000000000001e-05 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684125+0000", 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5707000000000001e-05 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884504+0000", 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200379487 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884519+0000", 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4953000000000001e-05 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885504+0000", 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098537600000000005 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885574+0000", 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0036999999999994e-05 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.180 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/32 ECSubWrite(tid=161, reqid=client.4728.0:127, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.861100+0000", 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.2940062880000001, 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0029967010000000001, 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861100+0000", 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861100+0000", 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861103+0000", 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.526e-06 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861466+0000", 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000363879 2024-09-16T04:41:47.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861515+0000", 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8631000000000002e-05 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861543+0000", 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8388999999999999e-05 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861938+0000", 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00039414099999999998 2024-09-16T04:41:47.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.861996+0000", 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8281e-05 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.864025+0000", 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020294000000000002 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.864097+0000", 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1453999999999997e-05 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.1807349469999999, 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20260964000000001, 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974373+0000", 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.683e-06 2024-09-16T04:41:47.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974841+0000", 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468255 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974891+0000", 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0198000000000003e-05 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.074965+0000", 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100073226 2024-09-16T04:41:47.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175170+0000", 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100205119 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175184+0000", 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4626e-05 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176898+0000", 2024-09-16T04:41:47.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017136390000000001 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176981+0000", 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2893999999999996e-05 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=16, reqid=client.4728.0:131, at_version=36'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.0800916570000001, 2024-09-16T04:41:47.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102541787, 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075016+0000", 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.06e-06 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075046+0000", 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0485999999999999e-05 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075066+0000", 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9572000000000002e-05 2024-09-16T04:41:47.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075078+0000", 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2626000000000001e-05 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176189+0000", 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10111062699999999 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176204+0000", 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4543e-05 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.177494+0000", 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012900450000000001 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.177556+0000", 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2828000000000002e-05 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s1 36/32 ECSubWrite(tid=17, reqid=client.4728.0:139, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.086841+0000", 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.0682655539999999, 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002770237, 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.086841+0000", 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.086841+0000", 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.086842+0000", 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.75e-06 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087176+0000", 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00033319000000000003 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087220+0000", 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4422000000000003e-05 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087236+0000", 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6308999999999999e-05 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087402+0000", 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016525399999999999 2024-09-16T04:41:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087430+0000", 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8785e-05 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.089530+0000", 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020995570000000002 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.089611+0000", 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0970000000000006e-05 2024-09-16T04:41:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=11, reqid=client.4728.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.087873+0000", 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.0672333279999999, 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101857478, 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087873+0000", 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087873+0000", 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087875+0000", 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.013e-06 2024-09-16T04:41:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088207+0000", 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00033206000000000002 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088233+0000", 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5885999999999999e-05 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088248+0000", 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5126000000000001e-05 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088401+0000", 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015341399999999999 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088416+0000", 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.435e-05 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189673+0000", 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101257364 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189730+0000", 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7265000000000001e-05 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=12, reqid=client.4728.0:134, at_version=36'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.088282+0000", 2024-09-16T04:41:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.066824451, 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102194618, 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088282+0000", 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088282+0000", 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088283+0000", 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8000000000000005e-07 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088303+0000", 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0801999999999998e-05 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088316+0000", 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088325+0000", 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189373+0000", 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10104791 2024-09-16T04:41:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189388+0000", 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4772e-05 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.190406+0000", 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001018337 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.190476+0000", 2024-09-16T04:41:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0177000000000003e-05 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=14, reqid=client.4728.0:148, at_version=36'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.297882+0000", 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.85722474299999996, 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101650159, 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297882+0000", 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297882+0000", 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297883+0000", 2024-09-16T04:41:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5400000000000001e-06 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297912+0000", 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8509000000000001e-05 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297928+0000", 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6226e-05 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398020+0000", 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100092311 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398158+0000", 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013751099999999999 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398185+0000", 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7129000000000001e-05 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.399451+0000", 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012658890000000001 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.399532+0000", 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1044000000000005e-05 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=14, reqid=client.4728.0:134, at_version=0'0, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:41:47.211 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.391878+0000", 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.76322821900000004, 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101508103, 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391878+0000", 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391878+0000", 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391880+0000", 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7489999999999999e-06 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391901+0000", 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1112999999999999e-05 2024-09-16T04:41:47.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391933+0000", 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2112e-05 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.492027+0000", 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100094133 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.492147+0000", 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011994600000000001 2024-09-16T04:41:47.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.492166+0000", 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8624999999999999e-05 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.493339+0000", 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001172827 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.493386+0000", 2024-09-16T04:41:47.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7598000000000001e-05 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=19, reqid=client.4728.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.398134+0000", 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.75697252500000001, 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023288670000000001, 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398134+0000", 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398134+0000", 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398135+0000", 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398164+0000", 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8467999999999999e-05 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398188+0000", 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4119000000000001e-05 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398210+0000", 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2693000000000001e-05 2024-09-16T04:41:47.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398372+0000", 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016191 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398388+0000", 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.522e-05 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.400389+0000", 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020012540000000001 2024-09-16T04:41:47.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.400463+0000", 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3732999999999994e-05 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:160, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.70144333699999994, 2024-09-16T04:41:47.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20191688999999999, 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453664+0000", 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453695+0000", 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0998999999999999e-05 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453714+0000", 2024-09-16T04:41:47.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9745000000000002e-05 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.553888+0000", 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10017351100000001 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654137+0000", 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100248949 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654151+0000", 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4279e-05 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655526+0000", 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013750520000000001 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655580+0000", 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3634999999999997e-05 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s1 36/32 ECSubWrite(tid=21, reqid=client.4728.0:156, at_version=36'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.554011+0000", 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.60109499300000002, 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102169199, 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.225 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554011+0000", 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554011+0000", 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554012+0000", 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0359999999999999e-06 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554201+0000", 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000188963 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554225+0000", 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4202e-05 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554239+0000", 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3246e-05 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654468+0000", 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10022879 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654496+0000", 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8204999999999998e-05 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.656122+0000", 2024-09-16T04:41:47.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016259930000000001 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.656180+0000", 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8764000000000002e-05 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s1 36/32 ECSubWrite(tid=22, reqid=client.4728.0:157, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:41:47.229 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.56974099499999997, 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102232006, 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585367+0000", 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585532+0000", 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164928 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585561+0000", 2024-09-16T04:41:47.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9391e-05 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585578+0000", 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6569999999999999e-05 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.685858+0000", 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100280496 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.685876+0000", 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7499999999999998e-05 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.687532+0000", 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016559509999999999 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.687597+0000", 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5456999999999993e-05 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.233 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s1 36/32 ECSubWrite(tid=23, reqid=client.4728.0:158, at_version=36'11, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.587811+0000", 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.56729531499999997, 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101093054, 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.587811+0000", 2024-09-16T04:41:47.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.587811+0000", 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.587813+0000", 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7659999999999999e-06 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.587844+0000", 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1328999999999998e-05 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.587873+0000", 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9298999999999998e-05 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.687956+0000", 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100083148 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.688050+0000", 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3966000000000001e-05 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.688064+0000", 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4005999999999999e-05 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.688835+0000", 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00077024899999999998 2024-09-16T04:41:47.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.688904+0000", 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9290999999999998e-05 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=17, reqid=client.4728.0:160, at_version=0'0, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.896258+0000", 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.25884816999999999, 2024-09-16T04:41:47.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10164221599999999, 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.896258+0000", 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.896258+0000", 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.896260+0000", 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.103e-06 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.896285+0000", 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4998999999999999e-05 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.896322+0000", 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6412000000000003e-05 2024-09-16T04:41:47.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996464+0000", 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100142192 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996603+0000", 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013938199999999999 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996620+0000", 2024-09-16T04:41:47.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6478999999999999e-05 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.997833+0000", 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012135550000000001 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.997900+0000", 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7094000000000005e-05 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.6s2 36/32 ECSubRead(tid=4, to_read={3:6877e850:::smithi16530038-5:head=0,20480,0}, subchunks={3:6877e850:::smithi16530038-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.996655+0000", 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.158451641, 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.074926758999999996, 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996655+0000", 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996655+0000", 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996656+0000", 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.573e-06 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996674+0000", 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7359000000000001e-05 2024-09-16T04:41:47.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996715+0000", 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1622000000000002e-05 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996743+0000", 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7875e-05 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.071211+0000", 2024-09-16T04:41:47.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.074467478000000004 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.071581+0000", 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037085200000000002 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=18, reqid=client.4728.0:167, at_version=36'11, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:41:47.246 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:47.065496+0000", 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.089610526999999995, 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033404870000000001, 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065496+0000", 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.247 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065496+0000", 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065497+0000", 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6670000000000001e-06 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065595+0000", 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7515000000000004e-05 2024-09-16T04:41:47.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065625+0000", 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0110000000000001e-05 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065642+0000", 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6568000000000001e-05 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065769+0000", 2024-09-16T04:41:47.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012777 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.065783+0000", 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3519e-05 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.067250+0000", 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001467225 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.068836+0000", 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015861130000000001 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.251 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:47.253 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 1 to smithi16530038-48 2024-09-16T04:41:47.253 INFO:tasks.rados.rados.0.smithi165.stdout:42: finishing write tid 3 to smithi16530038-42 2024-09-16T04:41:47.253 INFO:tasks.rados.rados.0.smithi165.stdout:42: oid 42 updating version 0 to 12 2024-09-16T04:41:47.253 INFO:tasks.rados.rados.0.smithi165.stdout:42: oid 42 version 12 is already newer than 11 2024-09-16T04:41:47.253 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T04:41:47.253 INFO:tasks.rados.rados.0.smithi165.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:41:47.253 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 2 to smithi16530038-48 2024-09-16T04:41:47.254 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 3 to smithi16530038-48 2024-09-16T04:41:47.254 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:41:47.254 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:47.248+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:47.272 INFO:tasks.rados.rados.0.smithi165.stdout:51: finishing copy_from racing read to smithi16530038-20 2024-09-16T04:41:47.299 INFO:tasks.rados.rados.0.smithi165.stdout:42: done (10 left) 2024-09-16T04:41:47.299 INFO:tasks.rados.rados.0.smithi165.stdout:47: done (9 left) 2024-09-16T04:41:47.299 INFO:tasks.rados.rados.0.smithi165.stdout:49: done (8 left) 2024-09-16T04:41:47.299 INFO:tasks.rados.rados.0.smithi165.stdout:61: done (7 left) 2024-09-16T04:41:47.299 INFO:tasks.rados.rados.0.smithi165.stdout:62: read oid 33 snap -1 2024-09-16T04:41:47.299 INFO:tasks.rados.rados.0.smithi165.stdout:62: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:41:47.299 INFO:tasks.rados.rados.0.smithi165.stdout:63: snap_create 2024-09-16T04:41:47.310 INFO:tasks.rados.rados.0.smithi165.stdout:55: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T04:41:47.354 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:47.353+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:47.446 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:47.447 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:47.447 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:41:47.447 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:47.447 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:41:47.455 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:47.453+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:47.473 INFO:tasks.rados.rados.0.smithi165.stdout:51: finishing copy_from to smithi16530038-20 2024-09-16T04:41:47.473 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T04:41:47.555 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:47.552+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:47.566 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:47.566 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:47.567 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:41:47.567 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:41:47.567 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:47.568 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 4 to smithi16530038-48 2024-09-16T04:41:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:48: oid 48 updating version 0 to 15 2024-09-16T04:41:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:48: oid 48 version 15 is already newer than 13 2024-09-16T04:41:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:48: oid 48 version 15 is already newer than 14 2024-09-16T04:41:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T04:41:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-16T04:41:47.574 INFO:tasks.rados.rados.0.smithi165.stdout:60: finishing copy_from racing read to smithi16530038-26 2024-09-16T04:41:47.581 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:41:47.656 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:47.653+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:47.708 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4356.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 20.306897842000001, 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31207237399999999, 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:47.709 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4356", 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.59:0/2607617674", 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:41:47.710 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398126+0000", 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0999999999999998e-06 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398137+0000", 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0253e-05 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398158+0000", 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1401999999999999e-05 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398176+0000", 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8552000000000001e-05 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597014+0000", 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19883803 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.712 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597043+0000", 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8683999999999998e-05 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.604764+0000", 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0077206829999999999 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605002+0000", 2024-09-16T04:41:47.713 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023771400000000001 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 15", 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605234+0000", 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023226099999999999 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.606579+0000", 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013450650000000001 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:47.714 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710048+0000", 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103468869 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710127+0000", 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9491000000000002e-05 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710197+0000", 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9270000000000006e-05 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.715 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_repop_reply(client.4356.0:1 2.3 e26/25 ondisk, result = 0)", 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:27.709812+0000", 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 19.995210409999999, 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00039452699999999999, 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.716 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.709812+0000", 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.709812+0000", 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.709813+0000", 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8729999999999999e-06 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.717 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.709823+0000", 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2960000000000004e-06 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.709853+0000", 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9958e-05 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.709869+0000", 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6022e-05 2024-09-16T04:41:47.718 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710021+0000", 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015206700000000001 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710037+0000", 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6271999999999999e-05 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710206+0000", 2024-09-16T04:41:47.719 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016903900000000001 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 36/32 ECSubWrite(tid=25, reqid=client.4728.0:115, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.774066+0000", 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.930955998, 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0025945709999999999, 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.720 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774066+0000", 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774066+0000", 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774068+0000", 2024-09-16T04:41:47.721 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0700000000000001e-06 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774386+0000", 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000317757 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774420+0000", 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3998000000000002e-05 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.722 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774437+0000", 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6988999999999999e-05 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774664+0000", 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022714100000000001 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774692+0000", 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8008999999999999e-05 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.723 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.776592+0000", 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019001300000000001 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.776661+0000", 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8477000000000002e-05 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.724 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 36/32 ECSubWrite(tid=26, reqid=client.4728.0:116, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.930541485, 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101615175, 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1999999999999998e-07 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774502+0000", 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0398999999999999e-05 2024-09-16T04:41:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774516+0000", 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3822000000000001e-05 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774525+0000", 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8269999999999993e-06 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.874737+0000", 2024-09-16T04:41:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10021129300000001 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.874752+0000", 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5415999999999999e-05 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.875998+0000", 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001246236 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.876096+0000", 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7361999999999994e-05 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 36/32 ECSubWrite(tid=26, reqid=client.4728.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.966685+0000", 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.7383369850000001, 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20155758700000001, 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966685+0000", 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966685+0000", 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966687+0000", 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1500000000000002e-06 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966710+0000", 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3272e-05 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966742+0000", 2024-09-16T04:41:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1275999999999997e-05 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.066872+0000", 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100129973 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.167193+0000", 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100321778 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.167220+0000", 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7002e-05 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.168171+0000", 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095042200000000003 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.168243+0000", 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1713999999999998e-05 2024-09-16T04:41:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s4 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.974506+0000", 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.730515866, 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003493374, 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974506+0000", 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974506+0000", 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974508+0000", 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9929999999999998e-06 2024-09-16T04:41:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.975135+0000", 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000626645 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.975166+0000", 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1615000000000001e-05 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.975183+0000", 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6875999999999999e-05 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.975359+0000", 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017619599999999999 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.975389+0000", 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9899e-05 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977931+0000", 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0025415059999999998 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.978000+0000", 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8644000000000002e-05 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s4 36/32 ECSubWrite(tid=16, reqid=client.4728.0:131, at_version=36'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.977372+0000", 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.7276499830000001, 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001401673, 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977372+0000", 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.739 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977372+0000", 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977374+0000", 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5600000000000001e-06 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977405+0000", 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1775e-05 2024-09-16T04:41:47.740 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977429+0000", 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3796000000000001e-05 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977444+0000", 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.505e-05 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.741 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977600+0000", 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000155463 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.977618+0000", 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8142999999999999e-05 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.978703+0000", 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010850689999999999 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.742 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.978774+0000", 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0816999999999996e-05 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 36/32 ECSubWrite(tid=27, reqid=client.4728.0:136, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.075497+0000", 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.6295251630000001, 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026607979999999998, 2024-09-16T04:41:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075497+0000", 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075497+0000", 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.744 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075499+0000", 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6199999999999999e-06 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075749+0000", 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024988700000000001 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075781+0000", 2024-09-16T04:41:47.745 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1472e-05 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075797+0000", 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6106000000000001e-05 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075958+0000", 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016105000000000001 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.746 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075985+0000", 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7412000000000001e-05 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.078071+0000", 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020852639999999999 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.078158+0000", 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6986999999999999e-05 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.747 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s3 36/32 ECSubRead(tid=28, to_read={3:8583b9f2:::smithi16530038-39:head=0,4096,0}, subchunks={3:8583b9f2:::smithi16530038-39:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.178605+0000", 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.526417189, 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00069336899999999999, 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.748 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178605+0000", 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178605+0000", 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178607+0000", 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0999999999999998e-06 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.749 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178626+0000", 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9032e-05 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178658+0000", 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2532e-05 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178677+0000", 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.863e-05 2024-09-16T04:41:47.750 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178856+0000", 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000178519 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.179298+0000", 2024-09-16T04:41:47.751 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00044255599999999998 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 36/32 ECSubWrite(tid=29, reqid=client.4728.0:137, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.178719+0000", 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.5263029769999998, 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10198995499999999, 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.752 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178719+0000", 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178719+0000", 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.753 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178720+0000", 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3570000000000001e-06 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178763+0000", 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2781000000000003e-05 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178788+0000", 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4559e-05 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.754 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178807+0000", 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8836000000000001e-05 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.279430+0000", 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100623613 2024-09-16T04:41:47.755 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.279445+0000", 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5133000000000001e-05 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.280624+0000", 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011788930000000001 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.756 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.280709+0000", 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4783000000000005e-05 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s5 36/32 ECSubWrite(tid=165, reqid=client.4728.0:150, at_version=36'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.291267+0000", 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.4137550779999999, 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002162125, 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.757 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291267+0000", 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291267+0000", 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.758 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291269+0000", 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1440000000000001e-06 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291562+0000", 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029276500000000001 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291629+0000", 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7120999999999997e-05 2024-09-16T04:41:47.759 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291648+0000", 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9485000000000001e-05 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291792+0000", 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000143258 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.291806+0000", 2024-09-16T04:41:47.760 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3813e-05 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.293368+0000", 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001562068 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.293429+0000", 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1470999999999995e-05 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.761 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s5 36/32 ECSubWrite(tid=166, reqid=client.4728.0:151, at_version=36'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.293926+0000", 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.411095633, 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019825450000000001, 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.293926+0000", 2024-09-16T04:41:47.762 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.293926+0000", 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.293928+0000", 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.763 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.294031+0000", 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000102709 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.294066+0000", 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5395000000000002e-05 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.294090+0000", 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4066e-05 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.764 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.294235+0000", 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000144914 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.294249+0000", 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4253000000000001e-05 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.295811+0000", 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001561745 2024-09-16T04:41:47.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.295909+0000", 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7886000000000004e-05 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s5 36/32 ECSubWrite(tid=167, reqid=client.4728.0:152, at_version=36'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.395802+0000", 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.309219616, 2024-09-16T04:41:47.766 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001380332, 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.395802+0000", 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.395802+0000", 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.767 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.395804+0000", 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.697e-06 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.395845+0000", 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0454999999999998e-05 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.768 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.395886+0000", 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1505000000000003e-05 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.395913+0000", 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6945e-05 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.396088+0000", 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000174663 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.396111+0000", 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3743e-05 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.397090+0000", 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00097887400000000002 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.397183+0000", 2024-09-16T04:41:47.770 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2449999999999997e-05 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s4 36/32 ECSubWrite(tid=19, reqid=client.4728.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.398283+0000", 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.306738878, 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013192970000000001, 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.771 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398283+0000", 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398283+0000", 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398285+0000", 2024-09-16T04:41:47.772 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.79e-06 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398314+0000", 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8588000000000001e-05 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398354+0000", 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0899000000000003e-05 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.773 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398374+0000", 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9723000000000001e-05 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398533+0000", 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015839000000000001 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.398561+0000", 2024-09-16T04:41:47.774 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8045e-05 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.399528+0000", 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00096777199999999999 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.399602+0000", 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4090000000000004e-05 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 36/32 ECSubWrite(tid=30, reqid=client.4728.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.569075+0000", 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.1359471619999999, 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10136287400000001, 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569075+0000", 2024-09-16T04:41:47.776 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569075+0000", 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569077+0000", 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2369999999999999e-06 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.777 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569106+0000", 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8909000000000001e-05 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569137+0000", 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1408000000000002e-05 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569161+0000", 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3620000000000001e-05 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.778 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.669453+0000", 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100292283 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.669468+0000", 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4345000000000001e-05 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.779 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.670392+0000", 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00092416000000000004 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.670438+0000", 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5911999999999997e-05 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.780 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s3 36/32 ECSubRead(tid=31, to_read={3:8c8bcb2a:::smithi16530038-46:head=0,4096,0}, subchunks={3:8c8bcb2a:::smithi16530038-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.569521+0000", 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.13550124, 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100790267, 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569521+0000", 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.781 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569521+0000", 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569522+0000", 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5799999999999999e-06 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569541+0000", 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8170000000000001e-05 2024-09-16T04:41:47.782 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569578+0000", 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.7701999999999999e-05 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569605+0000", 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6821999999999998e-05 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.783 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.669910+0000", 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100304489 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.670311+0000", 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040150400000000001 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s5 36/32 ECSubWrite(tid=168, reqid=client.4728.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=36'3))", 2024-09-16T04:41:47.784 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.599333+0000", 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.1056887639999999, 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001676287, 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599333+0000", 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.785 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599333+0000", 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599335+0000", 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9300000000000002e-06 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599363+0000", 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7566e-05 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.786 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599444+0000", 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1363e-05 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599470+0000", 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5423e-05 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599653+0000", 2024-09-16T04:41:47.787 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000183699 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599678+0000", 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4692e-05 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.600901+0000", 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012234049999999999 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.601010+0000", 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010820899999999999 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s3 36/32 ECSubRead(tid=32, to_read={3:803446a7:::smithi16530038-31:head=0,126976,0}, subchunks={3:803446a7:::smithi16530038-31:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.899114+0000", 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.80590844100000003, 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100996225, 2024-09-16T04:41:47.789 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899114+0000", 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899114+0000", 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899115+0000", 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4100000000000001e-06 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899130+0000", 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5209e-05 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899158+0000", 2024-09-16T04:41:47.791 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7704000000000001e-05 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899180+0000", 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2599000000000002e-05 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.999382+0000", 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100201049 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.792 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.000110+0000", 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00072825400000000001 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 37/32 ECSubWrite(tid=34, reqid=client.4728.0:169, at_version=37'11, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:47.572952+0000", 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.13207047499999999, 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001730075, 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:47.793 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.572952+0000", 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.572952+0000", 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:47.794 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.572955+0000", 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0639999999999998e-06 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.573018+0000", 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3657000000000004e-05 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.573051+0000", 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2662e-05 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.795 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.573071+0000", 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9942999999999999e-05 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.573226+0000", 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015547799999999999 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:47.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.573242+0000", 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5719e-05 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.574616+0000", 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013736810000000001 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.574682+0000", 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5870999999999993e-05 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.797 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.798 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.798 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:47.798 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:47.798 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:47.799 INFO:tasks.rados.rados.0.smithi165.stdout:60: finishing copy_from to smithi16530038-26 2024-09-16T04:41:47.799 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:41:47.800 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:41:47.801 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:47.796+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:47.900 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:47.896+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:47.929 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:47.929 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:47.929 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:41:47.929 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:47.943 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:41:47.977 INFO:tasks.rados.rados.0.smithi165.stdout:62: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:41:48.002 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:48.001+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:48.062 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:41:48.074 INFO:tasks.rados.rados.0.smithi165.stdout:48: done (8 left) 2024-09-16T04:41:48.075 INFO:tasks.rados.rados.0.smithi165.stdout:51: done (7 left) 2024-09-16T04:41:48.075 INFO:tasks.rados.rados.0.smithi165.stdout:55: done (6 left) 2024-09-16T04:41:48.075 INFO:tasks.rados.rados.0.smithi165.stdout:60: done (5 left) 2024-09-16T04:41:48.075 INFO:tasks.rados.rados.0.smithi165.stdout:62: done (4 left) 2024-09-16T04:41:48.075 INFO:tasks.rados.rados.0.smithi165.stdout:63: done (3 left) 2024-09-16T04:41:48.075 INFO:tasks.rados.rados.0.smithi165.stdout:64: copy_from oid 41 from oid 33 current snap is 1 2024-09-16T04:41:48.075 INFO:tasks.rados.rados.0.smithi165.stdout:65: read oid 4 snap -1 2024-09-16T04:41:48.076 INFO:tasks.rados.rados.0.smithi165.stdout:65: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-16T04:41:48.076 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 28 current snap is 1 2024-09-16T04:41:48.076 INFO:tasks.rados.rados.0.smithi165.stdout:66: seq_num 53 ranges {458752=524288,983040=376832} 2024-09-16T04:41:48.091 INFO:tasks.rados.rados.0.smithi165.stdout:66: writing smithi16530038-28 from 458752 to 983040 tid 1 2024-09-16T04:41:48.097 INFO:tasks.rados.rados.0.smithi165.stdout:66: writing smithi16530038-28 from 983040 to 1359872 tid 2 2024-09-16T04:41:48.097 INFO:tasks.rados.rados.0.smithi165.stdout:67: rollback oid 44 current snap is 1 2024-09-16T04:41:48.097 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 0 2024-09-16T04:41:48.097 INFO:tasks.rados.rados.0.smithi165.stdout:68: setattr oid 26 current snap is 1 2024-09-16T04:41:48.100 INFO:tasks.rados.rados.0.smithi165.stdout:69: copy_from oid 21 from oid 15 current snap is 1 2024-09-16T04:41:48.100 INFO:tasks.rados.rados.0.smithi165.stdout:70: read oid 39 snap 0 2024-09-16T04:41:48.100 INFO:tasks.rados.rados.0.smithi165.stdout:70: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T04:41:48.100 INFO:tasks.rados.rados.0.smithi165.stdout:71: copy_from oid 45 from oid 8 current snap is 1 2024-09-16T04:41:48.100 INFO:tasks.rados.rados.0.smithi165.stdout:72: rollback oid 49 current snap is 1 2024-09-16T04:41:48.100 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 0 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:73: read oid 12 snap 0 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:73: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:74: rmattr oid 42 current snap is 1 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:74: done (13 left) 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:75: copy_from oid 1 from oid 37 current snap is 1 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:76: read oid 3 snap -1 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:76: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout:77: copy_from oid 29 from oid 14 current snap is 1 2024-09-16T04:41:48.101 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.103 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:48.101+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:48.117 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:48.118 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:48.118 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:41:48.118 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:41:48.119 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:48.132 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:41:48.205 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:48.201+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=11, reqid=client.4728.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:41:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.1686886730000001, 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20239900399999999, 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087079+0000", 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5730000000000002e-06 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087392+0000", 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000313164 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087425+0000", 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2768999999999997e-05 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187523+0000", 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100097504 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187687+0000", 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164537 2024-09-16T04:41:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187711+0000", 2024-09-16T04:41:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4209000000000001e-05 2024-09-16T04:41:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289394+0000", 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10168234399999999 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289476+0000", 2024-09-16T04:41:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1903999999999994e-05 2024-09-16T04:41:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 36/32 ECSubWrite(tid=27, reqid=client.4728.0:136, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.162018+0000", 2024-09-16T04:41:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.0937477260000001, 2024-09-16T04:41:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10274949799999999, 2024-09-16T04:41:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162018+0000", 2024-09-16T04:41:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162018+0000", 2024-09-16T04:41:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162020+0000", 2024-09-16T04:41:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.187e-06 2024-09-16T04:41:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162297+0000", 2024-09-16T04:41:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027736200000000002 2024-09-16T04:41:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162328+0000", 2024-09-16T04:41:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1248999999999998e-05 2024-09-16T04:41:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262471+0000", 2024-09-16T04:41:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100142278 2024-09-16T04:41:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262628+0000", 2024-09-16T04:41:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000157584 2024-09-16T04:41:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262676+0000", 2024-09-16T04:41:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7924999999999999e-05 2024-09-16T04:41:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.264719+0000", 2024-09-16T04:41:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020430090000000001 2024-09-16T04:41:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.264767+0000", 2024-09-16T04:41:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7904000000000001e-05 2024-09-16T04:41:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=12, reqid=client.4728.0:134, at_version=36'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.187573+0000", 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.068192662, 2024-09-16T04:41:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102594399, 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187573+0000", 2024-09-16T04:41:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187573+0000", 2024-09-16T04:41:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187574+0000", 2024-09-16T04:41:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0300000000000001e-06 2024-09-16T04:41:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187604+0000", 2024-09-16T04:41:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0565999999999998e-05 2024-09-16T04:41:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187624+0000", 2024-09-16T04:41:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9899e-05 2024-09-16T04:41:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187636+0000", 2024-09-16T04:41:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2353e-05 2024-09-16T04:41:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.288921+0000", 2024-09-16T04:41:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101284807 2024-09-16T04:41:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.288944+0000", 2024-09-16T04:41:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2858999999999999e-05 2024-09-16T04:41:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.290094+0000", 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011501479999999999 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.290167+0000", 2024-09-16T04:41:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2737000000000005e-05 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s2 36/32 ECSubRead(tid=28, to_read={3:8583b9f2:::smithi16530038-39:head=0,4096,0}, subchunks={3:8583b9f2:::smithi16530038-39:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.262513+0000", 2024-09-16T04:41:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.9932519609999999, 2024-09-16T04:41:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001765443, 2024-09-16T04:41:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262513+0000", 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262513+0000", 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262515+0000", 2024-09-16T04:41:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3030000000000001e-06 2024-09-16T04:41:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262535+0000", 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0196000000000001e-05 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262572+0000", 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6820999999999999e-05 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262604+0000", 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2223e-05 2024-09-16T04:41:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.263792+0000", 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011879359999999999 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.264279+0000", 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00048696399999999999 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 36/32 ECSubWrite(tid=29, reqid=client.4728.0:137, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.262654+0000", 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.993110784, 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002628296, 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262654+0000", 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262654+0000", 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262656+0000", 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1400000000000001e-06 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262692+0000", 2024-09-16T04:41:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6865e-05 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262717+0000", 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4386e-05 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262735+0000", 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8113000000000001e-05 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.291 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.264352+0000", 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016174360000000001 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.264370+0000", 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7992e-05 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.265249+0000", 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00087846899999999999 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.265283+0000", 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3895e-05 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.as3 36/32 ECSubRead(tid=13, to_read={3:5619390f:::smithi16530038-41:head=0,4096,0}, subchunks={3:5619390f:::smithi16530038-41:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.9661919399999999, 2024-09-16T04:41:48.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20090377300000001, 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:41:48.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289575+0000", 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.717e-06 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289598+0000", 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3326e-05 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289642+0000", 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3174999999999999e-05 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.389821+0000", 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100179457 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490099+0000", 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10027802299999999 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490477+0000", 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037807500000000001 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=14, reqid=client.4728.0:134, at_version=0'0, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.391902+0000", 2024-09-16T04:41:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.863862994, 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10002425299999999, 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391902+0000", 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391902+0000", 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391904+0000", 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9369999999999998e-06 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391926+0000", 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2232e-05 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391960+0000", 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3764999999999999e-05 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.391979+0000", 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8703e-05 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490627+0000", 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.098647846999999997 2024-09-16T04:41:48.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490656+0000", 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9526000000000002e-05 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.491866+0000", 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012098020000000001 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.491927+0000", 2024-09-16T04:41:48.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0441e-05 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.as3 36/32 ECSubRead(tid=15, to_read={3:5f4281cc:::smithi16530038-45:head=0,4096,0}, subchunks={3:5f4281cc:::smithi16530038-45:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.393687+0000", 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.8620780379999999, 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097890126999999993, 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.302 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.393687+0000", 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.393687+0000", 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.393689+0000", 2024-09-16T04:41:48.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6869999999999999e-06 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.393705+0000", 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6495999999999999e-05 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.393736+0000", 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0988999999999998e-05 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.393754+0000", 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7282000000000002e-05 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.491220+0000", 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097465934000000004 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.491577+0000", 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00035773899999999998 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.305 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 36/32 ECSubWrite(tid=30, reqid=client.4728.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.568885+0000", 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.686880224, 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013529169999999999, 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.568885+0000", 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.568885+0000", 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.568887+0000", 2024-09-16T04:41:48.307 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.81e-06 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.568909+0000", 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2569e-05 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.568943+0000", 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3664999999999997e-05 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.308 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.568969+0000", 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6183e-05 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569120+0000", 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015046699999999999 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569140+0000", 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0593000000000001e-05 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.570195+0000", 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001055168 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.570238+0000", 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2462000000000001e-05 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.310 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s2 36/32 ECSubRead(tid=31, to_read={3:8c8bcb2a:::smithi16530038-46:head=0,4096,0}, subchunks={3:8c8bcb2a:::smithi16530038-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.569542+0000", 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.6862232000000001, 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00064750099999999998, 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569542+0000", 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569542+0000", 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569544+0000", 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6500000000000001e-06 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569561+0000", 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7119999999999999e-05 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569593+0000", 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2468000000000002e-05 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569643+0000", 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0158000000000003e-05 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.569803+0000", 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015924 2024-09-16T04:41:48.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.570190+0000", 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00038686500000000002 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=16, reqid=client.4728.0:171, at_version=36'9, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.897877+0000", 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3578879100000001, 2024-09-16T04:41:48.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00165552, 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.897877+0000", 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.897877+0000", 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.897879+0000", 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7e-06 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.897907+0000", 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8107e-05 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.897935+0000", 2024-09-16T04:41:48.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7390999999999999e-05 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.897950+0000", 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5065000000000001e-05 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.898149+0000", 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019974500000000001 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.898182+0000", 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2425000000000002e-05 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899483+0000", 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001300903 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.899533+0000", 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0183999999999999e-05 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.318 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s2 36/32 ECSubRead(tid=32, to_read={3:803446a7:::smithi16530038-31:head=0,126976,0}, subchunks={3:803446a7:::smithi16530038-31:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.996629+0000", 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.2591362400000001, 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00091328700000000004, 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996629+0000", 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996629+0000", 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996631+0000", 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4499999999999998e-06 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996652+0000", 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0052e-05 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996688+0000", 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6421999999999998e-05 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996708+0000", 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0162000000000001e-05 2024-09-16T04:41:48.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.996861+0000", 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015296400000000001 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.997542+0000", 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00068123700000000001 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 37/32 ECSubWrite(tid=17, reqid=client.4728.0:171, at_version=0'0, trim_to=0'0, roll_forward_to=36'9))", 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:47.299406+0000", 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.95635885600000003, 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014406390000000001, 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299406+0000", 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299406+0000", 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299407+0000", 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3300000000000003e-07 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299425+0000", 2024-09-16T04:41:48.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7975999999999999e-05 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299448+0000", 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3371999999999999e-05 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299465+0000", 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6833e-05 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299630+0000", 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164547 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.299660+0000", 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0451999999999999e-05 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.300753+0000", 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010931859999999999 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:47.300847+0000", 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3839999999999996e-05 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.327 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:48.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:41:48.329 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:41:48.330 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:48.328+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:48.380 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:41:48.380 INFO:tasks.rados.rados.0.smithi165.stdout:68: done (15 left) 2024-09-16T04:41:48.380 INFO:tasks.rados.rados.0.smithi165.stdout:78: rollback oid 11 current snap is 1 2024-09-16T04:41:48.380 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 0 2024-09-16T04:41:48.380 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.430 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:48.424+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:48.457 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:48.457 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:48.457 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:41:48.457 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:48.471 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:41:48.490 INFO:tasks.rados.rados.0.smithi165.stdout:71: finishing copy_from racing read to smithi16530038-45 2024-09-16T04:41:48.496 INFO:tasks.rados.rados.0.smithi165.stdout:64: finishing copy_from racing read to smithi16530038-41 2024-09-16T04:41:48.531 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:48.528+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:48.607 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 1 to smithi16530038-28 2024-09-16T04:41:48.607 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 2 to smithi16530038-28 2024-09-16T04:41:48.632 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:48.628+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:48.646 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:48.646 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:41:48.647 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:41:48.647 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:41:48.647 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:48.660 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:41:48.680 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:48.680 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:48.690 INFO:tasks.rados.rados.0.smithi165.stdout:77: finishing copy_from racing read to smithi16530038-29 2024-09-16T04:41:48.698 INFO:tasks.rados.rados.0.smithi165.stdout:71: finishing copy_from to smithi16530038-45 2024-09-16T04:41:48.698 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T04:41:48.698 INFO:tasks.rados.rados.0.smithi165.stdout:71: done (15 left) 2024-09-16T04:41:48.698 INFO:tasks.rados.rados.0.smithi165.stdout:79: rollback oid 22 current snap is 1 2024-09-16T04:41:48.699 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 0 2024-09-16T04:41:48.699 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.704 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:48.705 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:48.733 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:48.732+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:48.734 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:41:48.734 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:41:48.735 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-16T04:41:48.735 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 16 2024-09-16T04:41:48.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:48.737 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:48.738 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:48.791 INFO:tasks.rados.rados.0.smithi165.stdout:64: finishing copy_from to smithi16530038-41 2024-09-16T04:41:48.792 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 10 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T04:41:48.792 INFO:tasks.rados.rados.0.smithi165.stdout:64: done (15 left) 2024-09-16T04:41:48.792 INFO:tasks.rados.rados.0.smithi165.stdout:80: read oid 41 snap -1 2024-09-16T04:41:48.792 INFO:tasks.rados.rados.0.smithi165.stdout:80: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:41:48.792 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.792 INFO:tasks.rados.rados.0.smithi165.stdout:77: finishing copy_from to smithi16530038-29 2024-09-16T04:41:48.792 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T04:41:48.793 INFO:tasks.rados.rados.0.smithi165.stdout:77: done (15 left) 2024-09-16T04:41:48.793 INFO:tasks.rados.rados.0.smithi165.stdout:81: rollback oid 33 current snap is 1 2024-09-16T04:41:48.793 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 0 2024-09-16T04:41:48.793 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.794 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:41:48.794 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:41:48.794 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:41:48.794 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:41:48.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.794 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 24.414951508000001, 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1755266710000001, 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.795 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374332+0000", 2024-09-16T04:41:48.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8059999999999999e-06 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374345+0000", 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2529e-05 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374365+0000", 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0695000000000001e-05 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374383+0000", 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7569000000000001e-05 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.798 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374514+0000", 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013070099999999999 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374544+0000", 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0731000000000001e-05 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.343896+0000", 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.969351224 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344169+0000", 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027390100000000001 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344583+0000", 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041308600000000001 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.346047+0000", 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014642699999999999 2024-09-16T04:41:48.800 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549695+0000", 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203648463 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549791+0000", 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6137999999999993e-05 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549856+0000", 2024-09-16T04:41:48.801 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.168535329000001, 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51346886800000002, 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.802 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 111 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:41:48.803 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620747+0000", 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620757+0000", 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0316000000000001e-05 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.804 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620778+0000", 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620792+0000", 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4362000000000001e-05 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928510+0000", 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.307717717 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928629+0000", 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011919 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928822+0000", 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019246199999999999 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930635+0000", 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00181292 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134065+0000", 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20343018399999999 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134149+0000", 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4635000000000005e-05 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134214+0000", 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4973000000000003e-05 2024-09-16T04:41:48.807 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.168464965999998, 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51366009000000001, 2024-09-16T04:41:48.808 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 112 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620817+0000", 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621035+0000", 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021731100000000001 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621061+0000", 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6744999999999999e-05 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621082+0000", 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1061999999999999e-05 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.811 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929460+0000", 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.308377346 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929595+0000", 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013512600000000001 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929798+0000", 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020277199999999999 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.812 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932465+0000", 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002667862 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134319+0000", 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20185381599999999 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134407+0000", 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7894999999999998e-05 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.813 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134476+0000", 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8829000000000001e-05 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.168165023, 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51357863199999998, 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.814 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 113 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.815 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621118+0000", 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.689e-06 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621129+0000", 2024-09-16T04:41:48.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621164+0000", 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5367999999999997e-05 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.817 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621182+0000", 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8352e-05 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930689+0000", 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.309507015 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930800+0000", 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000110871 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931026+0000", 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000226293 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.933569+0000", 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002542786 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134566+0000", 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200996272 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134640+0000", 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4412000000000005e-05 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134694+0000", 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4524000000000002e-05 2024-09-16T04:41:48.820 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.168068276, 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51372108500000002, 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.821 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 114 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621214+0000", 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621225+0000", 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1348999999999999e-05 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621245+0000", 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9966000000000002e-05 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621259+0000", 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4102e-05 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931642+0000", 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31038274300000002 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931776+0000", 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133552 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931973+0000", 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197262 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.033893+0000", 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101919751 2024-09-16T04:41:48.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134774+0000", 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10088169600000001 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134847+0000", 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2309000000000002e-05 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134934+0000", 2024-09-16T04:41:48.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6924999999999999e-05 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.167996466999998, 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51387291300000004, 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.827 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 115 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621286+0000", 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.26e-06 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621492+0000", 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020643500000000001 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621517+0000", 2024-09-16T04:41:48.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621534+0000", 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6515999999999999e-05 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932515+0000", 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31098146799999998 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.830 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932631+0000", 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011636 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932814+0000", 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000182406 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134236+0000", 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201422618 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.831 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135029+0000", 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079253199999999998 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135102+0000", 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3286000000000003e-05 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135157+0000", 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5043000000000001e-05 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.832 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.167714649000001, 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.717172162, 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.833 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 116 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621568+0000", 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621606+0000", 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8167999999999998e-05 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621628+0000", 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1742e-05 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621643+0000", 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5696e-05 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235313+0000", 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61366968300000002 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235449+0000", 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136036 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235657+0000", 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207405 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237049+0000", 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001392552 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338582+0000", 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101532626 2024-09-16T04:41:48.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338670+0000", 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7888999999999998e-05 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338738+0000", 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8769000000000005e-05 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.167606659, 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.81671015999999996, 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 117 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621676+0000", 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4440000000000001e-06 2024-09-16T04:41:48.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621686+0000", 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0845e-05 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621706+0000", 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9168999999999999e-05 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621723+0000", 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7538999999999999e-05 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236241+0000", 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61451815600000004 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236372+0000", 2024-09-16T04:41:48.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013062300000000001 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236574+0000", 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020165600000000001 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237570+0000", 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099642999999999997 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438224+0000", 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20065369 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438317+0000", 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3624999999999999e-05 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438384+0000", 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6983000000000006e-05 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.844 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.845 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.845 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.845 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:41:48.845 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 14.494532681999999, 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.508075677, 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 138 2024-09-16T04:41:48.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294750+0000", 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294760+0000", 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294779+0000", 2024-09-16T04:41:48.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8941999999999999e-05 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294794+0000", 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5038999999999999e-05 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405306+0000", 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11051159300000001 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405426+0000", 2024-09-16T04:41:48.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011977 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405611+0000", 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000185217 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709561+0000", 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394976699999998 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802549+0000", 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092988137999999998 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802707+0000", 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015861799999999999 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802824+0000", 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116517 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 14.494462559, 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60756662299999997, 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 139 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.852 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294820+0000", 2024-09-16T04:41:48.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4330000000000001e-06 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294830+0000", 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0023e-05 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294850+0000", 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0622e-05 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294864+0000", 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4042e-05 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406110+0000", 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11124526 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406222+0000", 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111871 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406398+0000", 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017598000000000001 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709577+0000", 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30317971500000002 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902101+0000", 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19252423499999999 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902267+0000", 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016537499999999999 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902385+0000", 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011806700000000001 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.857 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:41:48.858 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 14.494370431, 2024-09-16T04:41:48.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60781196900000001, 2024-09-16T04:41:48.858 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.858 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.858 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.858 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.858 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 140 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294912+0000", 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.407e-06 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295040+0000", 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012836600000000001 2024-09-16T04:41:48.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295062+0000", 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1682e-05 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295083+0000", 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1109000000000001e-05 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507008+0000", 2024-09-16T04:41:48.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21192486699999999 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507149+0000", 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141042 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507369+0000", 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022046500000000001 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710298+0000", 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202928308 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902525+0000", 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.192226958 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902611+0000", 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6274999999999996e-05 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.863 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902722+0000", 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011149 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 14.494142849999999, 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70811914799999998, 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.864 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 141 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:41:48.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295140+0000", 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295152+0000", 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2053e-05 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295172+0000", 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0282e-05 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295191+0000", 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8759000000000001e-05 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508486+0000", 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.213295545 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508630+0000", 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014419899999999999 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508824+0000", 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019358600000000001 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710318+0000", 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20149381 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.002981+0000", 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.29266302500000002 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003144+0000", 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016330500000000001 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003257+0000", 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113064 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.829686122, 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51415646199999998, 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.870 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 159 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959597+0000", 2024-09-16T04:41:48.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9190000000000002e-06 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959609+0000", 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959629+0000", 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0035e-05 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959645+0000", 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5786000000000001e-05 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167529+0000", 2024-09-16T04:41:48.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.207883768 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167762+0000", 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023295499999999999 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.168134+0000", 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037181099999999999 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.270830+0000", 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102696683 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473585+0000", 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20275470700000001 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473677+0000", 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1772000000000002e-05 2024-09-16T04:41:48.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473751+0000", 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4228999999999997e-05 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.82960536, 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51420228400000001, 2024-09-16T04:41:48.877 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 160 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959677+0000", 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959687+0000", 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959711+0000", 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3811999999999998e-05 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959727+0000", 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169193+0000", 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209466241 2024-09-16T04:41:48.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169340+0000", 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000147279 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169545+0000", 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000205167 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374117+0000", 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20457175999999999 2024-09-16T04:41:48.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473763+0000", 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099646306000000004 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473828+0000", 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4046000000000003e-05 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473878+0000", 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0241000000000002e-05 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.829528080999999, 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51424688600000001, 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.883 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 161 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959754+0000", 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959920+0000", 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166088 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959944+0000", 2024-09-16T04:41:48.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959963+0000", 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9176000000000001e-05 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371049+0000", 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41108644500000002 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371185+0000", 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013619899999999999 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371395+0000", 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020912 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.377708+0000", 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006313173 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473891+0000", 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.096182904 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473949+0000", 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8016999999999997e-05 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474000+0000", 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.888 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.829212324, 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51552797100000003, 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.889 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 162 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960070+0000", 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960077+0000", 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6830000000000008e-06 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960091+0000", 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3653e-05 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960101+0000", 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7629999999999995e-06 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372554+0000", 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41245356599999999 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372785+0000", 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023023100000000001 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.373156+0000", 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000371601 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474010+0000", 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100854291 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475328+0000", 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001317304 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475477+0000", 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014954899999999999 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475596+0000", 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011913699999999999 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.894 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.829164096, 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51788103100000005, 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 163 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960118+0000", 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960124+0000", 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960134+0000", 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0115999999999999e-05 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960144+0000", 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:41:48.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374237+0000", 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41409377600000002 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374462+0000", 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022478600000000001 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374797+0000", 2024-09-16T04:41:48.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000334586 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474018+0000", 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099220947000000004 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477813+0000", 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003795171 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477921+0000", 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010822100000000001 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477998+0000", 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6662999999999994e-05 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:41:48.900 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.829124882, 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51822496799999995, 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 164 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960157+0000", 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7399999999999991e-07 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960241+0000", 2024-09-16T04:41:48.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3677999999999994e-05 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960256+0000", 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5206e-05 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960267+0000", 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1372999999999999e-05 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375650+0000", 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41538323100000002 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375885+0000", 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000234758 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.376293+0000", 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040777599999999999 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474024+0000", 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097731440000000003 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478113+0000", 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004088225 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478261+0000", 2024-09-16T04:41:48.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014825 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478381+0000", 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012005700000000001 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.828998809, 2024-09-16T04:41:48.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51862660500000002, 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 165 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960283+0000", 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960289+0000", 2024-09-16T04:41:48.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.49e-06 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960301+0000", 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2585999999999999e-05 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960309+0000", 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472635+0000", 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51232596699999999 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472771+0000", 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136243 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473033+0000", 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026155699999999999 2024-09-16T04:41:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474539+0000", 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015065 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478656+0000", 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041164319999999997 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:41:48.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478793+0000", 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013757299999999999 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478909+0000", 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011530400000000001 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:153 3.3s0 3:c31e3ec6:::smithi16530038-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:41:48.912 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.4996996770000002, 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.511692229, 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 153 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:41:48.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289583+0000", 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.04e-06 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289590+0000", 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4899999999999997e-06 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289610+0000", 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9814999999999999e-05 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289625+0000", 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5553000000000002e-05 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.296844+0000", 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072185369999999997 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297045+0000", 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020125199999999999 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599740+0000", 2024-09-16T04:41:48.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.302695037 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599866+0000", 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012644500000000001 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.801273+0000", 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20140706 2024-09-16T04:41:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:41:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:41:48.918 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:41:48.921 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:48.924 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:48.924 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:48.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:41:48.925 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:48.925 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 3 to smithi16530038-28 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:73: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:69: finishing copy_from racing read to smithi16530038-21 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:73: done (15 left) 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:82: read oid 25 snap 0 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:82: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 4 to smithi16530038-28 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:66: oid 28 updating version 0 to 10 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:66: oid 28 version 10 is already newer than 8 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:66: oid 28 version 10 is already newer than 9 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 10 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-09-16T04:41:48.926 INFO:tasks.rados.rados.0.smithi165.stdout:66: left oid 28 (ObjNum 53 snap 1 seq_num 53) 2024-09-16T04:41:48.927 INFO:tasks.rados.rados.0.smithi165.stdout:66: done (15 left) 2024-09-16T04:41:48.927 INFO:tasks.rados.rados.0.smithi165.stdout:83: read oid 13 snap -1 2024-09-16T04:41:48.927 INFO:tasks.rados.rados.0.smithi165.stdout:83: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:41:48.927 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.927 INFO:tasks.rados.rados.0.smithi165.stdout:70: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T04:41:48.927 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:48.928 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:48.921+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:48.937 INFO:tasks.rados.rados.0.smithi165.stdout:70: done (15 left) 2024-09-16T04:41:48.937 INFO:tasks.rados.rados.0.smithi165.stdout:84: read oid 16 snap -1 2024-09-16T04:41:48.937 INFO:tasks.rados.rados.0.smithi165.stdout:84: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-16T04:41:48.937 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:48.993 INFO:tasks.rados.rados.0.smithi165.stdout:79: finishing rollback tid 0 to smithi16530038-22 2024-09-16T04:41:49.016 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:49.016 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:49.022 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:49.019+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:49.049 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:49.049 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:49.057 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:49.058 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:41:49.060 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:49.060 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:49.077 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:41:49.077 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:49.077 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:49.081 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-09-16T04:41:48.992567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5685811,"num_objects":27,"num_object_clones":0,"num_object_copies":169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":176,"num_read_kb":1667,"num_write":252,"num_write_kb":8758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":137,"acting":137,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":446152,"kb_used_data":13640,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509503288,"statfs":{"total":1546188226560,"available":1545731366912,"internally_reserved":0,"allocated":13967360,"data_stored":12179367,"data_compressed":652473,"data_compressed_allocated":925696,"data_compressed_original":2711552,"omap_allocated":25522,"internal_metadata":442473550},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5095424,"num_objects":23,"num_object_clones":0,"num_object_copies":161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":42,"num_read_kb":1551,"num_write":52,"num_write_kb":5425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.282526"},"pg_stats":[{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"35'2","reported_seq":24,"reported_epoch":36,"state":"unknown","last_fresh":"2024-09-16T04:41:44.952133+0000","last_change":"2024-09-16T04:41:47.305695+0000","last_active":"2024-09-16T04:41:47.305695+0000","last_peered":"2024-09-16T04:41:44.952133+0000","last_clean":"2024-09-16T04:41:44.952133+0000","last_became_active":"2024-09-16T04:41:42.061226+0000","last_became_peered":"2024-09-16T04:41:42.061226+0000","last_unstale":"2024-09-16T04:41:44.952133+0000","last_undegraded":"2024-09-16T04:41:44.952133+0000","last_fullsized":"2024-09-16T04:41:44.952133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40960,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.728086+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:43.728086+0000","last_peered":"2024-09-16T04:41:43.728086+0000","last_clean":"2024-09-16T04:41:43.728086+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:43.728086+0000","last_undegraded":"2024-09-16T04:41:43.728086+0000","last_fullsized":"2024-09-16T04:41:43.728086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":24,"reported_epoch":36,"state":"peering","last_fresh":"2024-09-16T04:41:44.952133+0000","last_change":"2024-09-16T04:41:47.100786+0000","last_active":"2024-09-16T04:41:47.100786+0000","last_peered":"2024-09-16T04:41:44.952133+0000","last_clean":"2024-09-16T04:41:44.952133+0000","last_became_active":"2024-09-16T04:41:42.061226+0000","last_became_peered":"2024-09-16T04:41:42.061226+0000","last_unstale":"2024-09-16T04:41:44.952133+0000","last_undegraded":"2024-09-16T04:41:44.952133+0000","last_fullsized":"2024-09-16T04:41:44.952133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7,9,12,13,15],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.712647+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:43.712647+0000","last_peered":"2024-09-16T04:41:43.712647+0000","last_clean":"2024-09-16T04:41:43.712647+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:43.712647+0000","last_undegraded":"2024-09-16T04:41:43.712647+0000","last_fullsized":"2024-09-16T04:41:43.712647+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":19,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.233848+0000","last_change":"2024-09-16T04:41:42.063325+0000","last_active":"2024-09-16T04:41:44.233848+0000","last_peered":"2024-09-16T04:41:44.233848+0000","last_clean":"2024-09-16T04:41:44.233848+0000","last_became_active":"2024-09-16T04:41:42.061620+0000","last_became_peered":"2024-09-16T04:41:42.061620+0000","last_unstale":"2024-09-16T04:41:44.233848+0000","last_undegraded":"2024-09-16T04:41:44.233848+0000","last_fullsized":"2024-09-16T04:41:44.233848+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:03.789879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.795436+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:43.795436+0000","last_peered":"2024-09-16T04:41:43.795436+0000","last_clean":"2024-09-16T04:41:43.795436+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:43.795436+0000","last_undegraded":"2024-09-16T04:41:43.795436+0000","last_fullsized":"2024-09-16T04:41:43.795436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"35'2","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.933050+0000","last_change":"2024-09-16T04:41:42.020013+0000","last_active":"2024-09-16T04:41:43.933050+0000","last_peered":"2024-09-16T04:41:43.933050+0000","last_clean":"2024-09-16T04:41:43.933050+0000","last_became_active":"2024-09-16T04:41:41.818237+0000","last_became_peered":"2024-09-16T04:41:41.818237+0000","last_unstale":"2024-09-16T04:41:43.933050+0000","last_undegraded":"2024-09-16T04:41:43.933050+0000","last_fullsized":"2024-09-16T04:41:43.933050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:41.494998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49152,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.821078+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:43.821078+0000","last_peered":"2024-09-16T04:41:43.821078+0000","last_clean":"2024-09-16T04:41:43.821078+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:43.821078+0000","last_undegraded":"2024-09-16T04:41:43.821078+0000","last_fullsized":"2024-09-16T04:41:43.821078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"35'2","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.233412+0000","last_change":"2024-09-16T04:41:42.121505+0000","last_active":"2024-09-16T04:41:44.233412+0000","last_peered":"2024-09-16T04:41:44.233412+0000","last_clean":"2024-09-16T04:41:44.233412+0000","last_became_active":"2024-09-16T04:41:42.119957+0000","last_became_peered":"2024-09-16T04:41:42.119957+0000","last_unstale":"2024-09-16T04:41:44.233412+0000","last_undegraded":"2024-09-16T04:41:44.233412+0000","last_fullsized":"2024-09-16T04:41:44.233412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:09.270265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229376,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.720927+0000","last_change":"2024-09-16T04:41:42.022282+0000","last_active":"2024-09-16T04:41:44.720927+0000","last_peered":"2024-09-16T04:41:44.720927+0000","last_clean":"2024-09-16T04:41:44.720927+0000","last_became_active":"2024-09-16T04:41:41.819817+0000","last_became_peered":"2024-09-16T04:41:41.819817+0000","last_unstale":"2024-09-16T04:41:44.720927+0000","last_undegraded":"2024-09-16T04:41:44.720927+0000","last_fullsized":"2024-09-16T04:41:44.720927+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:03.678810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":35,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.027934+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:44.027934+0000","last_peered":"2024-09-16T04:41:44.027934+0000","last_clean":"2024-09-16T04:41:44.027934+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:44.027934+0000","last_undegraded":"2024-09-16T04:41:44.027934+0000","last_fullsized":"2024-09-16T04:41:44.027934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.819426+0000","last_change":"2024-09-16T04:41:41.812373+0000","last_active":"2024-09-16T04:41:43.819426+0000","last_peered":"2024-09-16T04:41:43.819426+0000","last_clean":"2024-09-16T04:41:43.819426+0000","last_became_active":"2024-09-16T04:41:41.811680+0000","last_became_peered":"2024-09-16T04:41:41.811680+0000","last_unstale":"2024-09-16T04:41:43.819426+0000","last_undegraded":"2024-09-16T04:41:43.819426+0000","last_fullsized":"2024-09-16T04:41:43.819426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:52.944066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.720745+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:41:44.720745+0000","last_peered":"2024-09-16T04:41:44.720745+0000","last_clean":"2024-09-16T04:41:44.720745+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:41:44.720745+0000","last_undegraded":"2024-09-16T04:41:44.720745+0000","last_fullsized":"2024-09-16T04:41:44.720745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.895334+0000","last_change":"2024-09-16T04:41:41.962201+0000","last_active":"2024-09-16T04:41:43.895334+0000","last_peered":"2024-09-16T04:41:43.895334+0000","last_clean":"2024-09-16T04:41:43.895334+0000","last_became_active":"2024-09-16T04:41:41.960834+0000","last_became_peered":"2024-09-16T04:41:41.960834+0000","last_unstale":"2024-09-16T04:41:43.895334+0000","last_undegraded":"2024-09-16T04:41:43.895334+0000","last_fullsized":"2024-09-16T04:41:43.895334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:02.598487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.821842+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:43.821842+0000","last_peered":"2024-09-16T04:41:43.821842+0000","last_clean":"2024-09-16T04:41:43.821842+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:43.821842+0000","last_undegraded":"2024-09-16T04:41:43.821842+0000","last_fullsized":"2024-09-16T04:41:43.821842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"35'2","reported_seq":25,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.831414+0000","last_change":"2024-09-16T04:41:42.158966+0000","last_active":"2024-09-16T04:41:44.831414+0000","last_peered":"2024-09-16T04:41:44.831414+0000","last_clean":"2024-09-16T04:41:44.831414+0000","last_became_active":"2024-09-16T04:41:41.957014+0000","last_became_peered":"2024-09-16T04:41:41.957014+0000","last_unstale":"2024-09-16T04:41:44.831414+0000","last_undegraded":"2024-09-16T04:41:44.831414+0000","last_fullsized":"2024-09-16T04:41:44.831414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:05.689742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311296,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.794955+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:43.794955+0000","last_peered":"2024-09-16T04:41:43.794955+0000","last_clean":"2024-09-16T04:41:43.794955+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:43.794955+0000","last_undegraded":"2024-09-16T04:41:43.794955+0000","last_fullsized":"2024-09-16T04:41:43.794955+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":229,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.720685+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:41:44.720685+0000","last_peered":"2024-09-16T04:41:44.720685+0000","last_clean":"2024-09-16T04:41:44.720685+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:41:44.720685+0000","last_undegraded":"2024-09-16T04:41:44.720685+0000","last_fullsized":"2024-09-16T04:41:44.720685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.795063+0000","last_change":"2024-09-16T04:41:42.215403+0000","last_active":"2024-09-16T04:41:43.795063+0000","last_peered":"2024-09-16T04:41:43.795063+0000","last_clean":"2024-09-16T04:41:43.795063+0000","last_became_active":"2024-09-16T04:41:42.013729+0000","last_became_peered":"2024-09-16T04:41:42.013729+0000","last_unstale":"2024-09-16T04:41:43.795063+0000","last_undegraded":"2024-09-16T04:41:43.795063+0000","last_fullsized":"2024-09-16T04:41:43.795063+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.639114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"35'4","reported_seq":25,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.136872+0000","last_change":"2024-09-16T04:41:42.152493+0000","last_active":"2024-09-16T04:41:44.136872+0000","last_peered":"2024-09-16T04:41:44.136872+0000","last_clean":"2024-09-16T04:41:44.136872+0000","last_became_active":"2024-09-16T04:41:42.051456+0000","last_became_peered":"2024-09-16T04:41:42.051456+0000","last_unstale":"2024-09-16T04:41:44.136872+0000","last_undegraded":"2024-09-16T04:41:44.136872+0000","last_fullsized":"2024-09-16T04:41:44.136872+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:52.942596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180224,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":4,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.a","version":"35'4","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.335135+0000","last_change":"2024-09-16T04:41:42.061337+0000","last_active":"2024-09-16T04:41:44.335135+0000","last_peered":"2024-09-16T04:41:44.335135+0000","last_clean":"2024-09-16T04:41:44.335135+0000","last_became_active":"2024-09-16T04:41:42.059056+0000","last_became_peered":"2024-09-16T04:41:42.059056+0000","last_unstale":"2024-09-16T04:41:44.335135+0000","last_undegraded":"2024-09-16T04:41:44.335135+0000","last_fullsized":"2024-09-16T04:41:44.335135+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:04.364555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":4,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"35'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:43.895369+0000","last_change":"2024-09-16T04:41:42.161718+0000","last_active":"2024-09-16T04:41:43.895369+0000","last_peered":"2024-09-16T04:41:43.895369+0000","last_clean":"2024-09-16T04:41:43.895369+0000","last_became_active":"2024-09-16T04:41:42.059548+0000","last_became_peered":"2024-09-16T04:41:42.059548+0000","last_unstale":"2024-09-16T04:41:43.895369+0000","last_undegraded":"2024-09-16T04:41:43.895369+0000","last_fullsized":"2024-09-16T04:41:43.895369+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:58.865572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"36'10","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:41:47.308468+0000","last_change":"2024-09-16T04:41:42.059491+0000","last_active":"2024-09-16T04:41:47.308468+0000","last_peered":"2024-09-16T04:41:47.308468+0000","last_clean":"2024-09-16T04:41:47.308468+0000","last_became_active":"2024-09-16T04:41:42.057674+0000","last_became_peered":"2024-09-16T04:41:42.057674+0000","last_unstale":"2024-09-16T04:41:47.308468+0000","last_undegraded":"2024-09-16T04:41:47.308468+0000","last_fullsized":"2024-09-16T04:41:47.308468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:05.738393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1818624,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":1542,"num_write":10,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"35'4","reported_seq":21,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.230575+0000","last_change":"2024-09-16T04:41:42.055455+0000","last_active":"2024-09-16T04:41:44.230575+0000","last_peered":"2024-09-16T04:41:44.230575+0000","last_clean":"2024-09-16T04:41:44.230575+0000","last_became_active":"2024-09-16T04:41:42.053696+0000","last_became_peered":"2024-09-16T04:41:42.053696+0000","last_unstale":"2024-09-16T04:41:44.230575+0000","last_undegraded":"2024-09-16T04:41:44.230575+0000","last_fullsized":"2024-09-16T04:41:44.230575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:07.867564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212992,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"35'4","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T04:41:44.231674+0000","last_change":"2024-09-16T04:41:42.459967+0000","last_active":"2024-09-16T04:41:44.231674+0000","last_peered":"2024-09-16T04:41:44.231674+0000","last_clean":"2024-09-16T04:41:44.231674+0000","last_became_active":"2024-09-16T04:41:42.158174+0000","last_became_peered":"2024-09-16T04:41:42.158174+0000","last_unstale":"2024-09-16T04:41:44.231674+0000","last_undegraded":"2024-09-16T04:41:44.231674+0000","last_fullsized":"2024-09-16T04:41:44.231674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:59.172130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"36'7","reported_seq":33,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:41:47.272123+0000","last_change":"2024-09-16T04:41:42.153067+0000","last_active":"2024-09-16T04:41:47.272123+0000","last_peered":"2024-09-16T04:41:47.272123+0000","last_clean":"2024-09-16T04:41:47.272123+0000","last_became_active":"2024-09-16T04:41:41.950892+0000","last_became_peered":"2024-09-16T04:41:41.950892+0000","last_unstale":"2024-09-16T04:41:47.272123+0000","last_undegraded":"2024-09-16T04:41:47.272123+0000","last_fullsized":"2024-09-16T04:41:47.272123+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:52.816938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1032192,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":5095424,"num_objects":23,"num_object_clones":0,"num_object_copies":161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":40,"num_read_kb":1549,"num_write":47,"num_write_kb":5296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11186176,"data_stored":11186176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49,"ondisk_log_size":49,"up":119,"acting":119,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247817,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28484,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343356,"statfs":{"total":96636764160,"available":96607596544,"internally_reserved":0,"allocated":1486848,"data_stored":1782040,"data_compressed":72020,"data_compressed_allocated":98304,"data_compressed_original":626688,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":675840,"data_stored":505970,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":724992,"data_stored":555122,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313224,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30180,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341660,"statfs":{"total":96636764160,"available":96605859840,"internally_reserved":0,"allocated":3231744,"data_stored":3059448,"data_compressed":40075,"data_compressed_allocated":57344,"data_compressed_original":114688,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313224,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29784,"kb_used_data":2752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342056,"statfs":{"total":96636764160,"available":96606265344,"internally_reserved":0,"allocated":2818048,"data_stored":2643685,"data_compressed":42996,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":397312,"data_stored":699080,"data_compressed":63256,"data_compressed_allocated":86016,"data_compressed_original":602112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280519,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":483328,"data_stored":313458,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":352256,"data_stored":180384,"data_compressed":34228,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":198770,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":335872,"data_stored":170152,"data_compressed":31311,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,8,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280519,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":790528,"data_stored":620658,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":385024,"data_stored":215154,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280519,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":348160,"data_stored":180365,"data_compressed":34228,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":491520,"data_stored":321650,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280519,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":663552,"data_stored":489605,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":413696,"data_stored":243826,"data_compressed":37151,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3194880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":946176,"data_stored":946176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":331776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":339968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":815104,"data_stored":815104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2478080,"data_stored":2478080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":299008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":122880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:49.083 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-09-16T04:41:49.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 16 2024-09-16T04:41:49.089 INFO:tasks.rados.rados.0.smithi165.stdout:69: finishing copy_from to smithi16530038-21 2024-09-16T04:41:49.089 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T04:41:49.089 INFO:tasks.rados.rados.0.smithi165.stdout:69: done (15 left) 2024-09-16T04:41:49.089 INFO:tasks.rados.rados.0.smithi165.stdout:85: read oid 20 snap -1 2024-09-16T04:41:49.089 INFO:tasks.rados.rados.0.smithi165.stdout:85: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T04:41:49.089 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:49.104 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:49.105 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:49.125 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:49.125+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:49.182 INFO:tasks.rados.rados.0.smithi165.stdout:84: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-16T04:41:49.204 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:49.204 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:49.205 INFO:tasks.rados.rados.0.smithi165.stdout:78: finishing rollback tid 0 to smithi16530038-11 2024-09-16T04:41:49.205 INFO:tasks.rados.rados.0.smithi165.stdout:78: finishing rollback tid 1 to smithi16530038-11 2024-09-16T04:41:49.205 INFO:tasks.rados.rados.0.smithi165.stdout:78: finishing rollback tid 2 to smithi16530038-11 2024-09-16T04:41:49.206 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 19 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T04:41:49.206 INFO:tasks.rados.rados.0.smithi165.stdout:81: finishing rollback tid 0 to smithi16530038-33 2024-09-16T04:41:49.206 INFO:tasks.rados.rados.0.smithi165.stdout:78: done (15 left) 2024-09-16T04:41:49.206 INFO:tasks.rados.rados.0.smithi165.stdout:84: done (14 left) 2024-09-16T04:41:49.206 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 42 current snap is 1 2024-09-16T04:41:49.206 INFO:tasks.rados.rados.0.smithi165.stdout:86: seq_num 54 ranges {704512=540672,1245184=376832} 2024-09-16T04:41:49.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:41:49.219 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:49.219 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:49.224 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:49.219+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:49.225 INFO:tasks.rados.rados.0.smithi165.stdout:86: writing smithi16530038-42 from 704512 to 1245184 tid 1 2024-09-16T04:41:49.232 INFO:tasks.rados.rados.0.smithi165.stdout:86: writing smithi16530038-42 from 1245184 to 1622016 tid 2 2024-09-16T04:41:49.232 INFO:tasks.rados.rados.0.smithi165.stdout:87: read oid 6 snap -1 2024-09-16T04:41:49.232 INFO:tasks.rados.rados.0.smithi165.stdout:87: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:41:49.232 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:49.236 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:49.237 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:49.257 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:49.258 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:49.275 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:49.276 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:49.325 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:49.324+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:49.385 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:49.385 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:49.395 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:49.395 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:49.400 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:41:49.402 INFO:tasks.rados.rados.0.smithi165.stdout:80: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:41:49.426 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:49.420+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:49.442 INFO:tasks.rados.rados.0.smithi165.stdout:80: done (15 left) 2024-09-16T04:41:49.442 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 32 current snap is 1 2024-09-16T04:41:49.442 INFO:tasks.rados.rados.0.smithi165.stdout:88: seq_num 55 ranges {524288=557056,1081344=376832} 2024-09-16T04:41:49.459 INFO:tasks.rados.rados.0.smithi165.stdout:88: writing smithi16530038-32 from 524288 to 1081344 tid 1 2024-09-16T04:41:49.466 INFO:tasks.rados.rados.0.smithi165.stdout:88: writing smithi16530038-32 from 1081344 to 1458176 tid 2 2024-09-16T04:41:49.466 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:49.521 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:49.521 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:49.527 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:49.525+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:49.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:41:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:83: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:41:49.605 INFO:tasks.rados.rados.0.smithi165.stdout:85: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:81: finishing rollback tid 1 to smithi16530038-33 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:81: finishing rollback tid 2 to smithi16530038-33 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 23 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:81: done (15 left) 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:83: done (14 left) 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:85: done (13 left) 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:89: rollback oid 20 current snap is 1 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 0 2024-09-16T04:41:49.619 INFO:tasks.rados.rados.0.smithi165.stdout:90: rollback oid 33 current snap is 1 2024-09-16T04:41:49.620 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 0 2024-09-16T04:41:49.620 INFO:tasks.rados.rados.0.smithi165.stdout:91: snap_create 2024-09-16T04:41:49.628 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:49.625+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:49.671 INFO:tasks.rados.rados.0.smithi165.stdout:88: finishing write tid 1 to smithi16530038-32 2024-09-16T04:41:49.702 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:49.702 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:49.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:41:49.729 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:49.728+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:49.799 INFO:tasks.rados.rados.0.smithi165.stdout:87: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:41:49.829 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:49.823+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:49.874 INFO:tasks.rados.rados.0.smithi165.stdout:88: finishing write tid 2 to smithi16530038-32 2024-09-16T04:41:49.887 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:49.887 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:49.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:41:49.902 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing rollback tid 0 to smithi16530038-20 2024-09-16T04:41:49.906 INFO:tasks.rados.rados.0.smithi165.stdout:82: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-16T04:41:49.949 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:49.949+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:50.004 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing rollback tid 1 to smithi16530038-20 2024-09-16T04:41:50.007 INFO:tasks.rados.rados.0.smithi165.stdout:86: finishing write tid 1 to smithi16530038-42 2024-09-16T04:41:50.032 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:50.028+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:50.048 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:50.048 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:50.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:41:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pg_num to 16 2024-09-16T04:41:50.075 INFO:tasks.rados.rados.0.smithi165.stdout:82: done (15 left) 2024-09-16T04:41:50.075 INFO:tasks.rados.rados.0.smithi165.stdout:87: done (14 left) 2024-09-16T04:41:50.075 INFO:tasks.rados.rados.0.smithi165.stdout:91: done (13 left) 2024-09-16T04:41:50.075 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 48 current snap is 2 2024-09-16T04:41:50.075 INFO:tasks.rados.rados.0.smithi165.stdout:92: seq_num 56 ranges {802816=573440,1376256=376832} 2024-09-16T04:41:50.097 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:41:50.098 INFO:tasks.rados.rados.0.smithi165.stdout:92: writing smithi16530038-48 from 802816 to 1376256 tid 1 2024-09-16T04:41:50.105 INFO:tasks.rados.rados.0.smithi165.stdout:92: writing smithi16530038-48 from 1376256 to 1753088 tid 2 2024-09-16T04:41:50.106 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 19 current snap is 2 2024-09-16T04:41:50.106 INFO:tasks.rados.rados.0.smithi165.stdout:93: seq_num 57 ranges {311296=589824,901120=376832} 2024-09-16T04:41:50.121 INFO:tasks.rados.rados.0.smithi165.stdout:93: writing smithi16530038-19 from 311296 to 901120 tid 1 2024-09-16T04:41:50.128 INFO:tasks.rados.rados.0.smithi165.stdout:93: writing smithi16530038-19 from 901120 to 1277952 tid 2 2024-09-16T04:41:50.128 INFO:tasks.rados.rados.0.smithi165.stdout:94: read oid 45 snap -1 2024-09-16T04:41:50.128 INFO:tasks.rados.rados.0.smithi165.stdout:94: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:41:50.128 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:50.128 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing rollback tid 2 to smithi16530038-20 2024-09-16T04:41:50.128 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 13 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T04:41:50.128 INFO:tasks.rados.rados.0.smithi165.stdout:89: done (15 left) 2024-09-16T04:41:50.129 INFO:tasks.rados.rados.0.smithi165.stdout:95: rollback oid 47 current snap is 2 2024-09-16T04:41:50.129 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 1 2024-09-16T04:41:50.129 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:50.133 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:50.127+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:50.177 INFO:tasks.rados.rados.0.smithi165.stdout:88: finishing write tid 3 to smithi16530038-32 2024-09-16T04:41:50.234 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:50.232+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:50.238 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:50.238 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:50.252 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:41:50.335 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:50.333+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:50.377 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:50.377 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:50.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:41:50.414 INFO:tasks.rados.rados.0.smithi165.stdout:86: finishing write tid 2 to smithi16530038-42 2024-09-16T04:41:50.436 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:50.431+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:50.482 INFO:tasks.rados.rados.0.smithi165.stdout:94: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:41:50.493 INFO:tasks.rados.rados.0.smithi165.stdout:94: done (15 left) 2024-09-16T04:41:50.493 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 38 current snap is 2 2024-09-16T04:41:50.493 INFO:tasks.rados.rados.0.smithi165.stdout:96: seq_num 58 ranges {622592=606208,1228800=376832} 2024-09-16T04:41:50.513 INFO:tasks.rados.rados.0.smithi165.stdout:96: writing smithi16530038-38 from 622592 to 1228800 tid 1 2024-09-16T04:41:50.520 INFO:tasks.rados.rados.0.smithi165.stdout:96: writing smithi16530038-38 from 1228800 to 1605632 tid 2 2024-09-16T04:41:50.520 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:88: finishing write tid 4 to smithi16530038-32 2024-09-16T04:41:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:88: oid 32 updating version 0 to 17 2024-09-16T04:41:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:88: oid 32 version 17 is already newer than 16 2024-09-16T04:41:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:88: oid 32 version 17 is already newer than 15 2024-09-16T04:41:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 17 (ObjNum 55 snap 1 seq_num 55) dirty exists 2024-09-16T04:41:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:88: left oid 32 (ObjNum 55 snap 1 seq_num 55) 2024-09-16T04:41:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:88: done (15 left) 2024-09-16T04:41:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:97: copy_from oid 21 from oid 46 current snap is 2 2024-09-16T04:41:50.532 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:50.536 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:50.533+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:50.561 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:50.561 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:50.575 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:41:50.637 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:50.632+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:50.738 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:50.736+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:50.749 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:50.749 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:50.763 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:41:50.823 INFO:tasks.rados.rados.0.smithi165.stdout:96: finishing write tid 1 to smithi16530038-38 2024-09-16T04:41:50.825 INFO:tasks.rados.rados.0.smithi165.stdout:96: finishing write tid 2 to smithi16530038-38 2024-09-16T04:41:50.831 INFO:tasks.rados.rados.0.smithi165.stdout:86: finishing write tid 3 to smithi16530038-42 2024-09-16T04:41:50.832 INFO:tasks.rados.rados.0.smithi165.stdout:90: finishing rollback tid 0 to smithi16530038-33 2024-09-16T04:41:50.839 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:50.836+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:50.887 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:50.887 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:50.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:41:50.946 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:50.943+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:51.024 INFO:tasks.rados.rados.0.smithi165.stdout:96: finishing write tid 3 to smithi16530038-38 2024-09-16T04:41:51.034 INFO:tasks.rados.rados.0.smithi165.stdout:90: finishing rollback tid 1 to smithi16530038-33 2024-09-16T04:41:51.034 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:51.034 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:51.035 INFO:tasks.rados.rados.0.smithi165.stdout:90: finishing rollback tid 2 to smithi16530038-33 2024-09-16T04:41:51.035 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 30 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T04:41:51.035 INFO:tasks.rados.rados.0.smithi165.stdout:90: done (15 left) 2024-09-16T04:41:51.035 INFO:tasks.rados.rados.0.smithi165.stdout:98: read oid 15 snap -1 2024-09-16T04:41:51.035 INFO:tasks.rados.rados.0.smithi165.stdout:98: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T04:41:51.035 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-16T04:41:51.040 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:51.036+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:51.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:41:51.089 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:41:51.115 INFO:tasks.rados.rados.0.smithi165.stdout:79: finishing rollback tid 1 to smithi16530038-22 2024-09-16T04:41:51.135 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 1 to smithi16530038-48 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:86: finishing write tid 4 to smithi16530038-42 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:86: oid 42 updating version 0 to 25 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:86: oid 42 updating version 25 to 27 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:86: oid 42 version 27 is already newer than 26 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 27 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:86: left oid 42 (ObjNum 54 snap 1 seq_num 54) 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:86: done (15 left) 2024-09-16T04:41:51.137 INFO:tasks.rados.rados.0.smithi165.stdout:99: snap_create 2024-09-16T04:41:51.141 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:51.141+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:51.160 INFO:tasks.rados.rados.0.smithi165.stdout:75: finishing copy_from racing read to smithi16530038-1 2024-09-16T04:41:51.216 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:51.216 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:51.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:41:51.242 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:51.241+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:51.343 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:51.340+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:51.399 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:51.399 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:51.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:41:51.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:41:51.445 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:51.440+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:51.537 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:51.537 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:51.545 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:51.544+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:51.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:41:51.560 INFO:tasks.rados.rados.0.smithi165.stdout:98: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T04:41:51.578 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:51.599 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:51.600 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:51.623 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:51.643 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:51.646 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:51.645+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:51.661 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 2 to smithi16530038-48 2024-09-16T04:41:51.662 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 3 to smithi16530038-48 2024-09-16T04:41:51.665 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 4 to smithi16530038-48 2024-09-16T04:41:51.665 INFO:tasks.rados.rados.0.smithi165.stdout:92: oid 48 updating version 0 to 33 2024-09-16T04:41:51.665 INFO:tasks.rados.rados.0.smithi165.stdout:92: oid 48 updating version 33 to 34 2024-09-16T04:41:51.665 INFO:tasks.rados.rados.0.smithi165.stdout:92: oid 48 version 34 is already newer than 32 2024-09-16T04:41:51.665 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 34 (ObjNum 56 snap 2 seq_num 56) dirty exists 2024-09-16T04:41:51.666 INFO:tasks.rados.rados.0.smithi165.stdout:92: left oid 48 (ObjNum 56 snap 2 seq_num 56) 2024-09-16T04:41:51.671 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:51.671 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:51.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:41:51.725 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:51.746 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:51.744+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:51.749 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:51.779 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:51.780 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:51.806 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:51.806 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:51.831 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:51.832 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:51.846 INFO:tasks.rados.rados.0.smithi165.stdout:96: finishing write tid 4 to smithi16530038-38 2024-09-16T04:41:51.846 INFO:tasks.rados.rados.0.smithi165.stdout:96: oid 38 updating version 0 to 9 2024-09-16T04:41:51.846 INFO:tasks.rados.rados.0.smithi165.stdout:96: oid 38 version 9 is already newer than 8 2024-09-16T04:41:51.846 INFO:tasks.rados.rados.0.smithi165.stdout:96: oid 38 updating version 9 to 10 2024-09-16T04:41:51.846 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 10 (ObjNum 58 snap 2 seq_num 58) dirty exists 2024-09-16T04:41:51.846 INFO:tasks.rados.rados.0.smithi165.stdout:96: left oid 38 (ObjNum 58 snap 2 seq_num 58) 2024-09-16T04:41:51.847 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:51.845+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:51.855 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/32 ECSubWriteReply(tid=16, last_complete=36'11, committed=1, applied=1))", 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.856543+0000", 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9954756719999995, 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29227929000000002, 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856543+0000", 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856543+0000", 2024-09-16T04:41:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856545+0000", 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.813e-06 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856554+0000", 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3100000000000006e-06 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856618+0000", 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4086999999999998e-05 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856633+0000", 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5003e-05 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.148792+0000", 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29215861399999998 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.148822+0000", 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0463e-05 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/32 ECSubWriteReply(tid=17, last_complete=36'12, committed=1, applied=1))", 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.856756+0000", 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9952627070000002, 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29225068999999998, 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856756+0000", 2024-09-16T04:41:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856756+0000", 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856757+0000", 2024-09-16T04:41:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.187e-06 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856764+0000", 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6930000000000001e-06 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856784+0000", 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9680000000000001e-05 2024-09-16T04:41:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.856795+0000", 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270000000000001e-05 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.148947+0000", 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29215226999999999 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.149007+0000", 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9589999999999997e-05 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:142 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6764599650000003, 2024-09-16T04:41:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98845964500000005, 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 142 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175562+0000", 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6839999999999998e-06 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175978+0000", 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041531000000000001 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176014+0000", 2024-09-16T04:41:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5913e-05 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176030+0000", 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5872999999999999e-05 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658368+0000", 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.48233846800000002 2024-09-16T04:41:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658643+0000", 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000274651 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.659966+0000", 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013230399999999999 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665031+0000", 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050652290000000001 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.163900+0000", 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.498868961 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164018+0000", 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118516 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/32 ECSubReadReply(tid=11, attrs_read=0))", 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.178369+0000", 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6736499360000003, 2024-09-16T04:41:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27394961400000001, 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178369+0000", 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178369+0000", 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178369+0000", 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8699999999999995e-07 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178385+0000", 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5546999999999999e-05 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178400+0000", 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5342999999999999e-05 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178408+0000", 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8269999999999998e-06 2024-09-16T04:41:51.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.452010+0000", 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27360215500000001 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.452318+0000", 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030815500000000001 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/32 ECSubReadReply(tid=13, attrs_read=0))", 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.178421+0000", 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.673598052, 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27432487, 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178421+0000", 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178421+0000", 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178421+0000", 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1300000000000001e-07 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178443+0000", 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2336000000000001e-05 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178454+0000", 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0207000000000001e-05 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178472+0000", 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8819000000000001e-05 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.452448+0000", 2024-09-16T04:41:51.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27397566000000001 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.452746+0000", 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029743499999999997 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.881 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/32 ECSubWriteReply(tid=16, last_complete=36'11, committed=1, applied=1))", 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.178468+0000", 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6735505829999999, 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27576574999999998, 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178468+0000", 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178468+0000", 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178471+0000", 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4030000000000001e-06 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178483+0000", 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2646999999999999e-05 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178516+0000", 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2555999999999998e-05 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178538+0000", 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2773999999999999e-05 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.453904+0000", 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.275366002 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.454234+0000", 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032936800000000002 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/32 ECSubWriteReply(tid=15, last_complete=36'10, committed=1, applied=1))", 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.178484+0000", 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6735350330000003, 2024-09-16T04:41:51.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27462976099999997, 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178484+0000", 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178484+0000", 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178484+0000", 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178488+0000", 2024-09-16T04:41:51.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9160000000000003e-06 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178498+0000", 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7040000000000006e-06 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178505+0000", 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000003e-06 2024-09-16T04:41:51.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.452840+0000", 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.274335473 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.453113+0000", 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000273078 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/32 ECSubWriteReply(tid=16, last_complete=36'11, committed=1, applied=1))", 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.178514+0000", 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6735046430000002, 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27530948599999999, 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.891 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178514+0000", 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178514+0000", 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178514+0000", 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.46e-07 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178517+0000", 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.977e-06 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178526+0000", 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5169999999999999e-06 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178533+0000", 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8430000000000004e-06 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.453794+0000", 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.2752617 2024-09-16T04:41:51.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.453824+0000", 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9102999999999999e-05 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/32 ECSubWriteReply(tid=17, last_complete=36'12, committed=1, applied=1))", 2024-09-16T04:41:51.895 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.178573+0000", 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6734454699999999, 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.275825822, 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178573+0000", 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178573+0000", 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178574+0000", 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8599999999999996e-07 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178580+0000", 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7130000000000002e-06 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178599+0000", 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8669999999999999e-05 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.178612+0000", 2024-09-16T04:41:51.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3117e-05 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.454342+0000", 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27573018300000002 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.454399+0000", 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7153e-05 2024-09-16T04:41:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:143 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6699816030000001, 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.982808864, 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:41:51.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182041+0000", 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5099999999999999e-06 2024-09-16T04:41:51.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182677+0000", 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063672200000000005 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182720+0000", 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2339999999999998e-05 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182742+0000", 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1929999999999998e-05 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661153+0000", 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.478411009 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661298+0000", 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145659 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.662610+0000", 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001311967 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766105+0000", 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10349428300000001 2024-09-16T04:41:51.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164694+0000", 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39858998099999998 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164846+0000", 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015146300000000001 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.906 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.5s4 36/32 ECSubWrite(tid=9, reqid=client.4728.0:119, at_version=0'0, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182394+0000", 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.669624786, 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.47454921999999999, 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182394+0000", 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182394+0000", 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182396+0000", 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0339999999999999e-06 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182425+0000", 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9306e-05 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182464+0000", 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8797000000000001e-05 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182491+0000", 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6492999999999999e-05 2024-09-16T04:41:51.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.554430+0000", 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37193904 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.554444+0000", 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4243000000000001e-05 2024-09-16T04:41:51.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.656888+0000", 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102443753 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.656943+0000", 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5553999999999997e-05 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.912 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:144 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6692417820000003, 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98240055400000004, 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:51.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 144 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182779+0000", 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182787+0000", 2024-09-16T04:41:51.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.977e-06 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182810+0000", 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3543000000000001e-05 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182839+0000", 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9e-05 2024-09-16T04:41:51.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663652+0000", 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.480813243 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663795+0000", 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143108 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.664540+0000", 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074409499999999996 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766192+0000", 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10165231600000001 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:51.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165013+0000", 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39882149300000003 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165177+0000", 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164146 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:145 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6691569189999997, 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.385587095, 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145 2024-09-16T04:41:51.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182863+0000", 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3039999999999999e-06 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182868+0000", 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182891+0000", 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2362999999999999e-05 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182911+0000", 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9813000000000002e-05 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665100+0000", 2024-09-16T04:41:51.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.482189849 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665304+0000", 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020365800000000001 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264646+0000", 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59934227699999998 2024-09-16T04:41:51.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264794+0000", 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147412 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.568449+0000", 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30365514599999999 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.925 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:1 3.ds0 3:bc30766e:::smithi16530038-11:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6396278850000003, 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0541568889999999, 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:51.926 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212394+0000", 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9000000000000002e-06 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212406+0000", 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2289999999999999e-05 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212428+0000", 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2410000000000001e-05 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212462+0000", 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3708999999999999e-05 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765550+0000", 2024-09-16T04:41:51.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55308802099999999 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765692+0000", 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014137199999999999 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.266548+0000", 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50085618700000001 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.930 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:2 3.ds0 3:b56b9d63:::smithi16530038-2:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.099304+0000", 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7527144210000003, 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.47165850399999998, 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.099304+0000", 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.099304+0000", 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.099308+0000", 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3830000000000001e-06 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.099323+0000", 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5440000000000001e-05 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.099354+0000", 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0732999999999999e-05 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.099374+0000", 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0254e-05 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.468872+0000", 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.36949830299999997 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469061+0000", 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000188176 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.570963+0000", 2024-09-16T04:41:51.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101902215 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:172 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.5528159490000002, 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57524237600000006, 2024-09-16T04:41:51.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 172 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:41:51.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299206+0000", 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6859999999999999e-06 2024-09-16T04:41:51.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299276+0000", 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9592999999999996e-05 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299323+0000", 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6749999999999998e-05 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299347+0000", 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4416000000000001e-05 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469676+0000", 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.170329287 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469835+0000", 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159006 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.874445+0000", 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40460963799999999 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:173 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.5526318610000001, 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57613007299999996, 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173 2024-09-16T04:41:51.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299389+0000", 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7099999999999999e-06 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299399+0000", 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0866000000000001e-05 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299427+0000", 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7117000000000001e-05 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299488+0000", 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1186000000000007e-05 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470324+0000", 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17083640999999999 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470484+0000", 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015948199999999999 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.875517+0000", 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40503330199999998 2024-09-16T04:41:51.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:174 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.5525030820000003, 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67607313300000005, 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.947 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299517+0000", 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.393e-06 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299526+0000", 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8230000000000001e-06 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299553+0000", 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6889999999999998e-05 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299573+0000", 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9944000000000001e-05 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471098+0000", 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.171524865 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471221+0000", 2024-09-16T04:41:51.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012369900000000001 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.975589+0000", 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50436751899999999 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.952 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.ds0 37/32 ECSubReadReply(tid=30, attrs_read=0))", 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.672680+0000", 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.1793389870000004, 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30294778100000003, 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.672680+0000", 2024-09-16T04:41:51.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.672680+0000", 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.672680+0000", 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.13e-07 2024-09-16T04:41:51.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.672837+0000", 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015678299999999999 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.672852+0000", 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5336e-05 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.672860+0000", 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9200000000000004e-06 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.974713+0000", 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30185221899999998 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.975627+0000", 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00091491000000000001 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.3.22:1 3.ds0 3:bfd7179d:::smithi16530038-33:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.180665+0000", 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.6713533570000001, 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30787368300000001, 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:51.957 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.180665+0000", 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.180665+0000", 2024-09-16T04:41:51.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.180669+0000", 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3830000000000001e-06 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.180683+0000", 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3900000000000001e-05 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.180709+0000", 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6650000000000001e-05 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.180738+0000", 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.906e-05 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:51.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.271693+0000", 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.090954873000000006 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.271891+0000", 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019746100000000001 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.488539+0000", 2024-09-16T04:41:51.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21664835599999999 2024-09-16T04:41:51.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:51.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:51.962 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:51.963 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:51.964 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:51.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:41:51.965 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:51.966 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:51.967 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:51.967 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:51.967 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:51.960+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:51.988 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:52.014 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:52.014 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:52.065 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:52.059+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:52.093 INFO:tasks.rados.rados.0.smithi165.stdout:92: done (15 left) 2024-09-16T04:41:52.094 INFO:tasks.rados.rados.0.smithi165.stdout:96: done (14 left) 2024-09-16T04:41:52.094 INFO:tasks.rados.rados.0.smithi165.stdout:98: done (13 left) 2024-09-16T04:41:52.094 INFO:tasks.rados.rados.0.smithi165.stdout:99: done (12 left) 2024-09-16T04:41:52.094 INFO:tasks.rados.rados.0.smithi165.stdout:100: snap_remove snap 2 2024-09-16T04:41:52.095 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:52.095 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:41:52.095 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:41:52.095 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:52.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:41:52.166 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:52.160+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:52.267 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:52.264+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:52.281 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:41:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:41:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:41:52.281 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:52.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:41:52.368 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:52.365+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:154 3.fs0 3:f6276c08:::smithi16530038-20:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.2110646139999997, 2024-09-16T04:41:52.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2626387860000001, 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 154 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:41:52.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210502+0000", 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9139999999999999e-06 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210519+0000", 2024-09-16T04:41:52.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6969999999999998e-05 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210555+0000", 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6099000000000001e-05 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210573+0000", 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7770000000000001e-05 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210738+0000", 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016486499999999999 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210962+0000", 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022410600000000001 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.269739+0000", 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.058777538 2024-09-16T04:41:52.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272567+0000", 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028276550000000001 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.472907+0000", 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20033962699999999 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.473136+0000", 2024-09-16T04:41:52.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000229242 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:155 3.fs0 3:f6276c08:::smithi16530038-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.2109486050000005, 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.061292586, 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.432 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 155 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210614+0000", 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7000000000000003e-07 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210619+0000", 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210636+0000", 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7436999999999999e-05 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210645+0000", 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.816e-06 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211299+0000", 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065386899999999996 2024-09-16T04:41:52.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211385+0000", 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5852000000000004e-05 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271332+0000", 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.05994666 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271584+0000", 2024-09-16T04:41:52.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025255399999999999 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271906+0000", 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032151100000000002 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:161 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~524288,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:52.437 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896315+0000", 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5252463880000002, 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41137774599999999, 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896315+0000", 2024-09-16T04:41:52.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896315+0000", 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896318+0000", 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896331+0000", 2024-09-16T04:41:52.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2877000000000001e-05 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896368+0000", 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7129000000000003e-05 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896395+0000", 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6429999999999999e-05 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.996662+0000", 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100267417 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.996840+0000", 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017823599999999999 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.307693+0000", 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.310852764 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:162 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~524288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896446+0000", 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5251155289999998, 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.412080487, 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.442 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 162 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896446+0000", 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896446+0000", 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896448+0000", 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896457+0000", 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3670000000000003e-06 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896479+0000", 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2183000000000001e-05 2024-09-16T04:41:52.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896496+0000", 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7220000000000001e-05 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.997292+0000", 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10079545600000001 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.997400+0000", 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108319 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.308527+0000", 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31112643899999998 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:163 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~524288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896528+0000", 2024-09-16T04:41:52.446 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5250334099999998, 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41302055999999998, 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 163 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896528+0000", 2024-09-16T04:41:52.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896528+0000", 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896530+0000", 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.333e-06 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896539+0000", 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3829999999999998e-06 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896566+0000", 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6543999999999999e-05 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896585+0000", 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9925999999999999e-05 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.997887+0000", 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101301393 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998027+0000", 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014029600000000001 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.309549+0000", 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31152168499999999 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:169 3.cs0 3:34145aa0:::smithi16530038-26:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:41:52.451 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5248849880000002, 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87942165800000005, 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 169 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:41:52.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896679+0000", 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3070000000000001e-06 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896692+0000", 2024-09-16T04:41:52.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2785999999999999e-05 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896719+0000", 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7540000000000001e-05 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896735+0000", 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5897e-05 2024-09-16T04:41:52.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998807+0000", 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102071833 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998930+0000", 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012282600000000001 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.572766+0000", 2024-09-16T04:41:52.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57383636000000005 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574732+0000", 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019661589999999999 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.775926+0000", 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20119392799999999 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.776098+0000", 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172022 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:170 3.cs0 3:34145aa0:::smithi16530038-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5247990680000001, 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67734535900000004, 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.457 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896764+0000", 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896768+0000", 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896780+0000", 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2003e-05 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896791+0000", 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0876000000000001e-05 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101155+0000", 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20436343900000001 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101252+0000", 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7195999999999996e-05 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573831+0000", 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.47257916700000002 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573966+0000", 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134816 2024-09-16T04:41:52.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574108+0000", 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014204199999999999 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/32 ECSubReadReply(tid=29, attrs_read=0))", 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.998266+0000", 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4232959039999997, 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103628948, 2024-09-16T04:41:52.462 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998266+0000", 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998266+0000", 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998268+0000", 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8369999999999999e-06 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998706+0000", 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043790000000000002 2024-09-16T04:41:52.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998728+0000", 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2002999999999999e-05 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.098827+0000", 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100099624 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101823+0000", 2024-09-16T04:41:52.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029955149999999998 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101895+0000", 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2069000000000004e-05 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.cs0 37/32 ECSubReadReply(tid=31, attrs_read=0))", 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.101665+0000", 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3198967709999998, 2024-09-16T04:41:52.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105066364, 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101665+0000", 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101665+0000", 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101665+0000", 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101665+0000", 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101696+0000", 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1517000000000003e-05 2024-09-16T04:41:52.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101739+0000", 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2858999999999997e-05 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.206624+0000", 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104885036 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.206731+0000", 2024-09-16T04:41:52.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106952 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/32 ECSubReadReply(tid=31, attrs_read=0))", 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.106018+0000", 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3155439690000001, 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20094226900000001, 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.106018+0000", 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.106018+0000", 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.106020+0000", 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9630000000000001e-06 2024-09-16T04:41:52.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.106310+0000", 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028988799999999997 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.106345+0000", 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5775999999999998e-05 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.106366+0000", 2024-09-16T04:41:52.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0755999999999999e-05 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.306880+0000", 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20051417699999999 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.306960+0000", 2024-09-16T04:41:52.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9709000000000005e-05 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/32 ECSubReadReply(tid=29, attrs_read=0))", 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.198379+0000", 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.2231830539999997, 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109346375, 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.198379+0000", 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.198379+0000", 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.198381+0000", 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1260000000000001e-06 2024-09-16T04:41:52.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.198877+0000", 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049576000000000004 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.198908+0000", 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1439999999999997e-05 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.198930+0000", 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1603e-05 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.306998+0000", 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.108068425 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.307725+0000", 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000727021 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 37/32 ECSubWriteReply(tid=32, last_complete=36'7, committed=1, applied=1))", 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.272178+0000", 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.1493836640000001, 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20037152999999999, 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272178+0000", 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272178+0000", 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272180+0000", 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.813e-06 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272191+0000", 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1029999999999999e-05 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272222+0000", 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0684e-05 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.372316+0000", 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10009447 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.472518+0000", 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020151300000001 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.472550+0000", 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2020000000000002e-05 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.481 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 37/32 ECSubWriteReply(tid=34, last_complete=37'11, committed=1, applied=1))", 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.575426+0000", 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8461356349999996, 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20110665799999999, 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.575426+0000", 2024-09-16T04:41:52.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.575426+0000", 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.575427+0000", 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4000000000000001e-07 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.575431+0000", 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7700000000000001e-06 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.575446+0000", 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4796e-05 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.675579+0000", 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100132244 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.775814+0000", 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10023552400000001 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.776533+0000", 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00071868400000000003 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.485 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 37/32 ECSubWriteReply(tid=35, last_complete=37'11, committed=1, applied=1))", 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.778055+0000", 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.6435063809999999, 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.200400681, 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778055+0000", 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778055+0000", 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778057+0000", 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778067+0000", 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7259999999999997e-06 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778096+0000", 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8827e-05 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.878191+0000", 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100094708 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.978403+0000", 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10021198100000001 2024-09-16T04:41:52.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.978456+0000", 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3418999999999998e-05 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 37/32 ECSubWriteReply(tid=35, last_complete=37'11, committed=1, applied=1))", 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.778342+0000", 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.643220017, 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.200511411, 2024-09-16T04:41:52.489 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778342+0000", 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778342+0000", 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778343+0000", 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.187e-06 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778349+0000", 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7030000000000003e-06 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.778369+0000", 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9939999999999999e-05 2024-09-16T04:41:52.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.878496+0000", 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10012726800000001 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.978833+0000", 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10033763 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.978853+0000", 2024-09-16T04:41:52.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9683000000000001e-05 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 37/32 ECSubWriteReply(tid=35, last_complete=37'11, committed=1, applied=1))", 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.878592+0000", 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.542970167, 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29452243700000003, 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.493 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.878592+0000", 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.878592+0000", 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.878593+0000", 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.234e-06 2024-09-16T04:41:52.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.878601+0000", 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3259999999999996e-06 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.878624+0000", 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2969999999999999e-05 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.978755+0000", 2024-09-16T04:41:52.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100131129 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.172978+0000", 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.194222859 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.173114+0000", 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135919 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:188 3.cs0 3:34145aa0:::smithi16530038-26:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.099752+0000", 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3218093240000002, 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27987911900000001, 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 188 2024-09-16T04:41:52.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099752+0000", 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099752+0000", 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099756+0000", 2024-09-16T04:41:52.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1970000000000001e-06 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099798+0000", 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2812999999999998e-05 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099833+0000", 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4425999999999999e-05 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099861+0000", 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8617000000000001e-05 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174508+0000", 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.074646951000000003 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174766+0000", 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025809800000000001 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.177119+0000", 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023529190000000002 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.279450+0000", 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102330465 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.379487+0000", 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100036718 2024-09-16T04:41:52.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.379632+0000", 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014491499999999999 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:207 3.fs0 3:f1175b03:::smithi16530038-29:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3211534189999998, 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.69161456600000004, 2024-09-16T04:41:52.502 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 207 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100411+0000", 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100423+0000", 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.202e-05 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100454+0000", 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0895999999999997e-05 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100469+0000", 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5614e-05 2024-09-16T04:41:52.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174514+0000", 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.074044911000000005 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174748+0000", 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023420200000000001 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.687971+0000", 2024-09-16T04:41:52.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51322260399999997 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690498+0000", 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002526978 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791786+0000", 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10128820500000001 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.792023+0000", 2024-09-16T04:41:52.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000236772 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:208 3.fs0 3:f1175b03:::smithi16530038-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3210630730000004, 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.58977269499999996, 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:52.508 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100500+0000", 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100507+0000", 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100524+0000", 2024-09-16T04:41:52.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.607e-05 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100532+0000", 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8270000000000004e-06 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274070+0000", 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17353716399999999 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:41:52.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274196+0000", 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012600600000000001 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689716+0000", 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41552020499999998 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689979+0000", 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026336099999999999 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690271+0000", 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029222200000000002 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.11s1 38/37 ECSubWrite(tid=46, reqid=client.4728.0:214, at_version=38'14, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.889132+0000", 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5324295660000002, 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.196942846, 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.513 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.889132+0000", 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.889132+0000", 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.889134+0000", 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.08e-06 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.889180+0000", 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5599000000000002e-05 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.889225+0000", 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4919999999999997e-05 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.889249+0000", 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4260000000000002e-05 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.079793+0000", 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.190543981 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.079817+0000", 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3672999999999998e-05 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.085999+0000", 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061825229999999997 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.086075+0000", 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5809999999999994e-05 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.517 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:52.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:41:52.519 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:52.515+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:52.620 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:52.616+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:52.692 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:52.692 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:41:52.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.692 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.10s2 42/41 ECSubWrite(tid=33, reqid=client.4728.0:187, at_version=42'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:52.692 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:41:52.692 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.175907076, 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17594979499999999, 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515487+0000", 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1999999999999998e-07 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515532+0000", 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4295999999999997e-05 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515554+0000", 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2325999999999999e-05 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615627+0000", 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100073459 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.19s4 42/41 ECSubWrite(tid=36, reqid=client.4728.0:263, at_version=42'14, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:52.695 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.075679689999999994, 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.075811044999999994, 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "no flag points reached", 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8699999999999995e-07 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615747+0000", 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2240999999999998e-05 2024-09-16T04:41:52.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615757+0000", 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0781000000000001e-05 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-09-16T04:41:52.698 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:52.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:41:52.721 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:52.717+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:52.821 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:52.815+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:52.830 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:41:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:41:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:41:52.830 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:52.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:41:52.922 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:52.920+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4356.0:1 2.3 e26/25 2:c4c92e5a:::rbd_trash:head v 26'1, mlcod=0'0)", 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:27.604703+0000", 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.402875134999999, 2024-09-16T04:41:53.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104050821, 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.604703+0000", 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.604703+0000", 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.604704+0000", 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9399999999999995e-07 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.604710+0000", 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8900000000000001e-06 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.604735+0000", 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4586000000000001e-05 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.704890+0000", 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100155173 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.705059+0000", 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000168825 2024-09-16T04:41:53.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.705120+0000", 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0976e-05 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.708637+0000", 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035167639999999999 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:27.708754+0000", 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117113 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 36/32 ECSubWriteReply(tid=9, last_complete=36'6, committed=1, applied=1))", 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.283388+0000", 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.7241897779999995, 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101412211, 2024-09-16T04:41:53.018 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.019 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.019 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.019 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.283388+0000", 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.283388+0000", 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.283390+0000", 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.283400+0000", 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7070000000000004e-06 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.283429+0000", 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8963e-05 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.283450+0000", 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0907000000000002e-05 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.384772+0000", 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101322305 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.384801+0000", 2024-09-16T04:41:53.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8493e-05 2024-09-16T04:41:53.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=10, attrs_read=0))", 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.7232410890000001, 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30135611200000001, 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284378+0000", 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1134000000000003e-05 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284392+0000", 2024-09-16T04:41:53.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3556e-05 2024-09-16T04:41:53.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585581+0000", 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30118932599999998 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585693+0000", 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011209600000000001 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=8, attrs_read=0))", 2024-09-16T04:41:53.028 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.288190+0000", 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.7193877610000001, 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29726268900000002, 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288190+0000", 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288190+0000", 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288193+0000", 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2840000000000001e-06 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288223+0000", 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0358999999999999e-05 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288257+0000", 2024-09-16T04:41:53.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4236999999999998e-05 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288279+0000", 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2033000000000001e-05 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585381+0000", 2024-09-16T04:41:53.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29710208399999999 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585453+0000", 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1692000000000004e-05 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=10, attrs_read=0))", 2024-09-16T04:41:53.033 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.289624+0000", 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.7179537939999996, 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.296969174, 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289624+0000", 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289624+0000", 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289626+0000", 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.756e-06 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289650+0000", 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3734e-05 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289692+0000", 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1811999999999999e-05 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289713+0000", 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1523e-05 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.586506+0000", 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.296792526 2024-09-16T04:41:53.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.586594+0000", 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7823000000000003e-05 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 36/32 ECSubWriteReply(tid=9, last_complete=36'6, committed=1, applied=1))", 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.289945+0000", 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.717633717, 2024-09-16T04:41:53.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29522546100000002, 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289945+0000", 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289945+0000", 2024-09-16T04:41:53.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289946+0000", 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0300000000000001e-06 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289952+0000", 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6100000000000002e-06 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289972+0000", 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.003e-05 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289983+0000", 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1090000000000001e-05 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585142+0000", 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29515883500000001 2024-09-16T04:41:53.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585170+0000", 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7866000000000001e-05 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.0s2 36/32 ECSubWrite(tid=19, reqid=client.4728.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.378774+0000", 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.6288038279999997, 2024-09-16T04:41:53.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101872857, 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.378774+0000", 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.378774+0000", 2024-09-16T04:41:53.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.378776+0000", 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.916e-06 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.378797+0000", 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0987000000000001e-05 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.378827+0000", 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9938999999999999e-05 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.378843+0000", 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.588e-05 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379000+0000", 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156389 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379024+0000", 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4162999999999999e-05 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.480607+0000", 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101583097 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.480647+0000", 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0485999999999998e-05 2024-09-16T04:41:53.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.0s2 36/32 ECSubRead(tid=20, to_read={3:0b497652:::smithi16530038-44:head=0,4096,0}, subchunks={3:0b497652:::smithi16530038-44:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.379839+0000", 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.6277387900000004, 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100760183, 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379839+0000", 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379839+0000", 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379841+0000", 2024-09-16T04:41:53.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6199999999999999e-06 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379859+0000", 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7819999999999999e-05 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379888+0000", 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9516e-05 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.379907+0000", 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8433000000000001e-05 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.480186+0000", 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100278963 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.480600+0000", 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041383099999999998 2024-09-16T04:41:53.049 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 36/32 ECSubWriteReply(tid=9, last_complete=36'6, committed=1, applied=1))", 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.380095+0000", 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.6274832149999998, 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10486598599999999, 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380095+0000", 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380095+0000", 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380104+0000", 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.363e-06 2024-09-16T04:41:53.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380113+0000", 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8130000000000002e-06 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380134+0000", 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0757000000000001e-05 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380146+0000", 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2418999999999999e-05 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.484928+0000", 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10478201700000001 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.484961+0000", 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2617000000000003e-05 2024-09-16T04:41:53.053 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=8, attrs_read=0))", 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.6273974090000003, 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.206073801, 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:41:53.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380199+0000", 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8009999999999999e-05 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380213+0000", 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3966e-05 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380221+0000", 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2169999999999997e-06 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585758+0000", 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.205537782 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.586255+0000", 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049623600000000005 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=10, attrs_read=0))", 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.380234+0000", 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.6273439759999997, 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27750757599999998, 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380234+0000", 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380234+0000", 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380235+0000", 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7599999999999997e-07 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380248+0000", 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.314e-05 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380258+0000", 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7929999999999992e-06 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380268+0000", 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9299999999999998e-06 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.657255+0000", 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.276987121 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.657742+0000", 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000487116 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.9s4 36/32 ECSubWrite(tid=21, reqid=client.4728.0:156, at_version=36'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.485608+0000", 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.5219700810000001, 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102682948, 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.485608+0000", 2024-09-16T04:41:53.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.485608+0000", 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.485610+0000", 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.317e-06 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.485781+0000", 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000170982 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.485806+0000", 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4663000000000001e-05 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.485821+0000", 2024-09-16T04:41:53.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4963999999999999e-05 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585823+0000", 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100002034 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585851+0000", 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8370000000000001e-05 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.588207+0000", 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023552960000000002 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.588291+0000", 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4321999999999997e-05 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.9s4 36/32 ECSubWrite(tid=25, reqid=client.4728.0:158, at_version=0'0, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.002206+0000", 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0053721290000004, 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102988417, 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.002206+0000", 2024-09-16T04:41:53.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.002206+0000", 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.002208+0000", 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1330000000000002e-06 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.002233+0000", 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.512e-05 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.002265+0000", 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1894000000000003e-05 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.002282+0000", 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6685999999999999e-05 2024-09-16T04:41:53.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.102515+0000", 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100233404 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.102546+0000", 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0490000000000001e-05 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.105195+0000", 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00264869 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:3 3.5s0 3:a54da884:::smithi16530038-14:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8311185029999999, 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50899456499999995, 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.071 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176463+0000", 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176482+0000", 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8975999999999999e-05 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176519+0000", 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6916999999999999e-05 2024-09-16T04:41:53.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.276624+0000", 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100105124 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478009+0000", 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.201384806 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:53.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478333+0000", 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000324381 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.685454+0000", 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20712119400000001 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 38/37 ECSubReadReply(tid=11, attrs_read=0))", 2024-09-16T04:41:53.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.578841+0000", 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4287373570000002, 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099902049000000007, 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.578841+0000", 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.578841+0000", 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.578843+0000", 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.747e-06 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.579108+0000", 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026556799999999999 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.579133+0000", 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5130000000000002e-05 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.579148+0000", 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4263e-05 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.678676+0000", 2024-09-16T04:41:53.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099527941999999994 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.678743+0000", 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7398999999999996e-05 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "replica_scrub(pg: 3.3s3,from:0'0,to:36'3,epoch:38/37,start:3:c0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.788461+0000", 2024-09-16T04:41:53.079 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.219116885, 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100598774, 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.788461+0000", 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.788461+0000", 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.788463+0000", 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.77e-06 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.788474+0000", 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0513000000000001e-05 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.788519+0000", 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5739999999999999e-05 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.888638+0000", 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10011824800000001 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.888854+0000", 2024-09-16T04:41:53.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021660200000000001 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.888895+0000", 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1216e-05 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.889060+0000", 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164685 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.8s5 39/37 ECSubWrite(tid=6, reqid=client.4728.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9172923590000002, 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.187882937, 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4300000000000003e-07 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090310+0000", 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3459999999999999e-05 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090342+0000", 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1973e-05 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090369+0000", 2024-09-16T04:41:53.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7250000000000002e-05 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.177958+0000", 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.087588497000000001 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.177971+0000", 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3869999999999999e-05 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.278092+0000", 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100120098 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.278169+0000", 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7245999999999999e-05 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.088 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.10s2 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.409445+0000", 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.59813314900000003, 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20474215100000001, 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409445+0000", 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409445+0000", 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409446+0000", 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.384e-06 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409492+0000", 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5126000000000001e-05 2024-09-16T04:41:53.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409533+0000", 2024-09-16T04:41:53.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0952999999999999e-05 2024-09-16T04:41:53.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409551+0000", 2024-09-16T04:41:53.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8666000000000001e-05 2024-09-16T04:41:53.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.512744+0000", 2024-09-16T04:41:53.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103193255 2024-09-16T04:41:53.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.512773+0000", 2024-09-16T04:41:53.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8615999999999998e-05 2024-09-16T04:41:53.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.614155+0000", 2024-09-16T04:41:53.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10138161 2024-09-16T04:41:53.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.614187+0000", 2024-09-16T04:41:53.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2540999999999999e-05 2024-09-16T04:41:53.095 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.095 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.095 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.096 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.10s2 42/41 ECSubWrite(tid=33, reqid=client.4728.0:187, at_version=42'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:53.096 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:41:53.096 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.49209180200000002, 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.203467801, 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515487+0000", 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1999999999999998e-07 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515532+0000", 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4295999999999997e-05 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515554+0000", 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2325999999999999e-05 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615627+0000", 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100073459 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.715775+0000", 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10014733300000001 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.715784+0000", 2024-09-16T04:41:53.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.718921+0000", 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031364079999999998 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.718954+0000", 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3612999999999998e-05 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.101 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.19s4 42/41 ECSubWrite(tid=36, reqid=client.4728.0:263, at_version=42'14, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.39186441599999999, 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101579977, 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:53.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615714+0000", 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8699999999999995e-07 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615747+0000", 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2240999999999998e-05 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615757+0000", 2024-09-16T04:41:53.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0781000000000001e-05 2024-09-16T04:41:53.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.715839+0000", 2024-09-16T04:41:53.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100081665 2024-09-16T04:41:53.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.715969+0000", 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012977099999999999 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.715981+0000", 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1902000000000001e-05 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.717252+0000", 2024-09-16T04:41:53.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012716940000000001 2024-09-16T04:41:53.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:53.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:53.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:53.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.717294+0000", 2024-09-16T04:41:53.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1436000000000001e-05 2024-09-16T04:41:53.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:53.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:53.108 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:53.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:41:53.111 INFO:tasks.rados.rados.0.smithi165.stdout:100: done (12 left) 2024-09-16T04:41:53.111 INFO:tasks.rados.rados.0.smithi165.stdout:101: read oid 6 snap 1 2024-09-16T04:41:53.111 INFO:tasks.rados.rados.0.smithi165.stdout:101: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:102: read oid 36 snap -1 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:102: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:103: copy_from oid 42 from oid 50 current snap is 3 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:79: finishing rollback tid 2 to smithi16530038-22 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 14 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:52: finishing write tid 4 to smithi16530038-27 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:52: oid 27 updating version 0 to 9 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:52: oid 27 updating version 9 to 11 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:52: oid 27 version 11 is already newer than 10 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 11 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:52: left oid 27 (ObjNum 50 snap 0 seq_num 50) 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:97: finishing copy_from racing read to smithi16530038-21 2024-09-16T04:41:53.112 INFO:tasks.rados.rados.0.smithi165.stdout:72: finishing rollback tid 0 to smithi16530038-49 2024-09-16T04:41:53.113 INFO:tasks.rados.rados.0.smithi165.stdout:72: finishing rollback tid 1 to smithi16530038-49 2024-09-16T04:41:53.113 INFO:tasks.rados.rados.0.smithi165.stdout:57: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:41:53.113 INFO:tasks.rados.rados.0.smithi165.stdout:65: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-16T04:41:53.114 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:53.108+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:53.114 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing rollback tid 0 to smithi16530038-47 2024-09-16T04:41:53.114 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing rollback tid 1 to smithi16530038-47 2024-09-16T04:41:53.114 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing rollback tid 2 to smithi16530038-47 2024-09-16T04:41:53.114 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T04:41:53.114 INFO:tasks.rados.rados.0.smithi165.stdout:97: finishing copy_from to smithi16530038-21 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 10 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:67: finishing rollback tid 0 to smithi16530038-44 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:67: finishing rollback tid 1 to smithi16530038-44 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:67: finishing rollback tid 2 to smithi16530038-44 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 8 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:72: finishing rollback tid 2 to smithi16530038-49 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 14 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:41:53.115 INFO:tasks.rados.rados.0.smithi165.stdout:52: done (14 left) 2024-09-16T04:41:53.116 INFO:tasks.rados.rados.0.smithi165.stdout:57: done (13 left) 2024-09-16T04:41:53.116 INFO:tasks.rados.rados.0.smithi165.stdout:65: done (12 left) 2024-09-16T04:41:53.116 INFO:tasks.rados.rados.0.smithi165.stdout:67: done (11 left) 2024-09-16T04:41:53.116 INFO:tasks.rados.rados.0.smithi165.stdout:72: done (10 left) 2024-09-16T04:41:53.116 INFO:tasks.rados.rados.0.smithi165.stdout:79: done (9 left) 2024-09-16T04:41:53.116 INFO:tasks.rados.rados.0.smithi165.stdout:95: done (8 left) 2024-09-16T04:41:53.117 INFO:tasks.rados.rados.0.smithi165.stdout:97: done (7 left) 2024-09-16T04:41:53.117 INFO:tasks.rados.rados.0.smithi165.stdout:104: rmattr oid 43 current snap is 3 2024-09-16T04:41:53.117 INFO:tasks.rados.rados.0.smithi165.stdout:104: done (7 left) 2024-09-16T04:41:53.117 INFO:tasks.rados.rados.0.smithi165.stdout:105: snap_remove snap 0 2024-09-16T04:41:53.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:41:53.211 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:53.208+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:53.248 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:53.248 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:53.248 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:41:53.248 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:53.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:41:53.311 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:53.307+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:53.388 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:53.389 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:53.389 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:41:53.389 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:41:53.389 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:53.403 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:41:53.412 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:53.408+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:53.513 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:53.508+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s4 36/32 ECSubWrite(tid=8, reqid=client.4728.0:109, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.1569470650000007, 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201785732, 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.533 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374507+0000", 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374649+0000", 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014185599999999999 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374672+0000", 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3277000000000001e-05 2024-09-16T04:41:53.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374685+0000", 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2797e-05 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574940+0000", 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20025501600000001 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574956+0000", 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5936000000000001e-05 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576258+0000", 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001302388 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576291+0000", 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.537 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 36/32 ECSubWrite(tid=22, reqid=client.4728.0:100, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.460943+0000", 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.0705091650000007, 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.11472982900000001, 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.538 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.460943+0000", 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.460943+0000", 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.460945+0000", 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.144e-06 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.461173+0000", 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00022797200000000001 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.461205+0000", 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1952000000000002e-05 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.461221+0000", 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6316000000000001e-05 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.561390+0000", 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016884299999999 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.561403+0000", 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2875e-05 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.575644+0000", 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014241701000000001 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.575673+0000", 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9026e-05 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 36/32 ECSubWrite(tid=23, reqid=client.4728.0:101, at_version=36'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.463990+0000", 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.0674626620000005, 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.11171482200000001, 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.463990+0000", 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.463990+0000", 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.463991+0000", 2024-09-16T04:41:53.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.009e-06 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.464012+0000", 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1271999999999999e-05 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.464033+0000", 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0992000000000002e-05 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.464046+0000", 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3206e-05 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.562065+0000", 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098018658999999994 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.562076+0000", 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0531e-05 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.575692+0000", 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013616009 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.575705+0000", 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3144e-05 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s4 36/32 ECSubWrite(tid=9, reqid=client.4728.0:110, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.475792+0000", 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.0556605779999995, 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10278398800000001, 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.548 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.475792+0000", 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.475792+0000", 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.475793+0000", 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0299999999999997e-07 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.475816+0000", 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2668999999999999e-05 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.475833+0000", 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7456999999999999e-05 2024-09-16T04:41:53.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.475844+0000", 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1051e-05 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.575506+0000", 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.099661845999999998 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.575513+0000", 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5980000000000002e-06 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.578542+0000", 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030292299999999999 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.578576+0000", 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4233999999999998e-05 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 36/32 ECSubWrite(tid=4, reqid=client.4728.0:112, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.9578407379999998, 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20173359499999999, 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573614+0000", 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.114e-06 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574099+0000", 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048458300000000002 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574141+0000", 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2551e-05 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574168+0000", 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6899000000000001e-05 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674350+0000", 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100181464 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674362+0000", 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2138e-05 2024-09-16T04:41:53.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775305+0000", 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100943171 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775345+0000", 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0674999999999999e-05 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 36/32 ECSubWrite(tid=5, reqid=client.4728.0:113, at_version=36'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.673261+0000", 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.8581916840000003, 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102794101, 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.558 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.673261+0000", 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.673261+0000", 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.673263+0000", 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.807e-06 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.673294+0000", 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1726000000000001e-05 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.673325+0000", 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0825999999999999e-05 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.673341+0000", 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5792000000000001e-05 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.774942+0000", 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10160104 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.774956+0000", 2024-09-16T04:41:53.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3873000000000001e-05 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775984+0000", 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010280039999999999 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.776055+0000", 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1032999999999996e-05 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s4 36/32 ECSubWrite(tid=13, reqid=client.4728.0:147, at_version=36'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.296002+0000", 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.2354502610000004, 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10336583100000001, 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296002+0000", 2024-09-16T04:41:53.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296002+0000", 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296004+0000", 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6470000000000001e-06 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296188+0000", 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000184162 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296238+0000", 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9497999999999999e-05 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296254+0000", 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5874000000000001e-05 2024-09-16T04:41:53.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296338+0000", 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4029000000000005e-05 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296346+0000", 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6119999999999999e-06 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.399325+0000", 2024-09-16T04:41:53.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102978692 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.399368+0000", 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3316999999999999e-05 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:164 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:41:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.6339183039999998, 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40676020400000001, 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 164 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:41:53.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897538+0000", 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0500000000000002e-06 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897554+0000", 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5886e-05 2024-09-16T04:41:53.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897591+0000", 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6899e-05 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897617+0000", 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5922e-05 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897727+0000", 2024-09-16T04:41:53.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011011200000000001 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897835+0000", 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107889 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304263+0000", 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40642834999999999 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304295+0000", 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1096000000000002e-05 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:165 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.6337786159999999, 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20425574899999999, 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.573 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 165 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897676+0000", 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897685+0000", 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.183e-06 2024-09-16T04:41:53.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897709+0000", 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4216e-05 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997840+0000", 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131417 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098015+0000", 2024-09-16T04:41:53.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100174894 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098124+0000", 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108714 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101885+0000", 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037611459999999999 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101930+0000", 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4675999999999997e-05 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:166 3.6s0 3.a17ee16 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.5335623529999998, 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.306539124, 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.578 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 166 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997892+0000", 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997905+0000", 2024-09-16T04:41:53.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2583e-05 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997926+0000", 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.101e-05 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098022+0000", 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100096478 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304408+0000", 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20638583099999999 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304429+0000", 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1132e-05 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.cs2 36/32 ECSubRead(tid=30, to_read={3:372a9655:::smithi16530038-32:head=0,131072,0}, subchunks={3:372a9655:::smithi16530038-32:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:53.582 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:47.098249+0000", 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.4332033979999999, 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200837185, 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098249+0000", 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098249+0000", 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098250+0000", 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4300000000000003e-07 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098256+0000", 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3899999999999998e-06 2024-09-16T04:41:53.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098270+0000", 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3766e-05 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198426+0000", 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100155832 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.298779+0000", 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10035303600000001 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.299086+0000", 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00030761800000000001 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.9s2 36/32 ECSubRead(tid=26, to_read={3:9de467d5:::smithi16530038-27:head=0,4096,0}, subchunks={3:9de467d5:::smithi16530038-27:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:47.198627+0000", 2024-09-16T04:41:53.586 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.3328256810000001, 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10696509, 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198627+0000", 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198627+0000", 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198628+0000", 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.13e-07 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198637+0000", 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7699999999999996e-06 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198661+0000", 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4165e-05 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198674+0000", 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.29e-05 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.305550+0000", 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.106876006 2024-09-16T04:41:53.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.305592+0000", 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1635999999999999e-05 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 38/32 ECSubWrite(tid=34, reqid=client.4728.0:182, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.2545446379999996, 2024-09-16T04:41:53.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103857901, 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276910+0000", 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277140+0000", 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00022952199999999999 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277172+0000", 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1958000000000001e-05 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277195+0000", 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3915999999999999e-05 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277528+0000", 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00033253800000000001 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277543+0000", 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5366000000000002e-05 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.380721+0000", 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103177899 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.380766+0000", 2024-09-16T04:41:53.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4648999999999999e-05 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds4 38/32 ECSubWrite(tid=34, reqid=client.4728.0:211, at_version=38'19, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.575466+0000", 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9559863459999995, 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103336706, 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.595 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.575466+0000", 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.575466+0000", 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.575467+0000", 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.575486+0000", 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9213000000000001e-05 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.575500+0000", 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3416e-05 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.675613+0000", 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100113681 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.676649+0000", 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010360339999999999 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.676674+0000", 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4822999999999999e-05 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.678706+0000", 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00203192 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.678803+0000", 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6968999999999999e-05 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.599 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.16s0 42/41 ECSubRead(tid=6, to_read={3:6877e850:::smithi16530038-5:head=0,20480,0}, subchunks={3:6877e850:::smithi16530038-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2232292870000001, 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20348143299999999, 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308260+0000", 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6318e-05 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308312+0000", 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2611999999999997e-05 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511485+0000", 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20317243800000001 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511705+0000", 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220065 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:41:53.603 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.222553454, 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20303517200000001, 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308901+0000", 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308970+0000", 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9240999999999997e-05 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309005+0000", 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4314999999999999e-05 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309023+0000", 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8879999999999999e-05 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511826+0000", 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202802133 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511934+0000", 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108756 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2223186619999999, 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20302847199999999, 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.607 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309135+0000", 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309242+0000", 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106959 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309260+0000", 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:41:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309270+0000", 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0339e-05 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512036+0000", 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20276591199999999 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512162+0000", 2024-09-16T04:41:53.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126675 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.406461+0000", 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.124992094, 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.105985461, 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.406461+0000", 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.406461+0000", 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.406463+0000", 2024-09-16T04:41:53.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0870000000000001e-06 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.406641+0000", 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177913 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.406678+0000", 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7135999999999998e-05 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.406693+0000", 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5109000000000001e-05 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512392+0000", 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.105699695 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512446+0000", 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3520999999999998e-05 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.614 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.19s2 42/41 ECSubWrite(tid=37, reqid=client.4728.0:264, at_version=42'15, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.518104+0000", 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.013348519, 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10411026, 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.518104+0000", 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.518104+0000", 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.518105+0000", 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9299999999999996e-07 2024-09-16T04:41:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.518147+0000", 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1842999999999999e-05 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.518168+0000", 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1549e-05 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.518187+0000", 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8899e-05 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.619806+0000", 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10161909600000001 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.619837+0000", 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0249000000000001e-05 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.622117+0000", 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0022803139999999999 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.622214+0000", 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7417000000000006e-05 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.619 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.14s4 42/41 ECSubWrite(tid=23, reqid=client.4728.0:197, at_version=42'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.520005+0000", 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.011447534, 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10317681099999999, 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.520005+0000", 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.520005+0000", 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.520007+0000", 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.520055+0000", 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8341999999999999e-05 2024-09-16T04:41:53.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.520090+0000", 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5361999999999998e-05 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.620196+0000", 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100105555 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.620347+0000", 2024-09-16T04:41:53.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015079199999999999 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.620371+0000", 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3998999999999998e-05 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623083+0000", 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002712622 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:53.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623182+0000", 2024-09-16T04:41:53.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8542999999999995e-05 2024-09-16T04:41:53.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.624 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.624 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.624 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:53.626 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:41:53.626 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:53.624+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:53.727 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:53.723+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:53.762 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:53.762 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:41:53.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDPGPushReply(3.17s0 43/41 [PushReplyOp(3:ed82f61a:::smithi16530038-1:2)])", 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.565285+0000", 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.19732415, 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.197360223, 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.565285+0000", 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.565285+0000", 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.565285+0000", 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9600000000000001e-07 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.565287+0000", 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6000000000000001e-06 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.565292+0000", 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.532e-06 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.565297+0000", 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7949999999999998e-06 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.765 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDPGPushReply(3.17s0 43/41 [PushReplyOp(3:ed82f61a:::smithi16530038-1:2)])", 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.097728037000000004, 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.097847122999999994, 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664919+0000", 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7880999999999999e-05 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664938+0000", 2024-09-16T04:41:53.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9510000000000001e-05 2024-09-16T04:41:53.769 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.769 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:53.769 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.769 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:53.769 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:41:53.769 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 2 2024-09-16T04:41:53.769 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:53.777 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:41:53.828 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:53.823+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:53.897 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:53.898 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:53.898 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:41:53.898 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:41:53.898 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:53.912 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:41:53.929 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:53.924+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:54.030 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:54.027+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 36/32 ECSubWrite(tid=7, reqid=client.4728.0:119, at_version=36'6, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.775234+0000", 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.2622506599999994, 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20113699500000001, 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.039 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775234+0000", 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775234+0000", 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.040 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775234+0000", 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4000000000000002e-07 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775251+0000", 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6843000000000001e-05 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775263+0000", 2024-09-16T04:41:54.041 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2091e-05 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.875361+0000", 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10009797199999999 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.975538+0000", 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10017714 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.042 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.975551+0000", 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.234e-05 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.976337+0000", 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00078631199999999995 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.043 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.976371+0000", 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3757000000000002e-05 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:136 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.061430+0000", 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.9760547439999998, 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30731443400000003, 2024-09-16T04:41:54.044 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 136 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.061430+0000", 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.045 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.061430+0000", 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.061431+0000", 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7880000000000001e-06 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062256+0000", 2024-09-16T04:41:54.046 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00082428099999999995 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062278+0000", 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2642000000000001e-05 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062290+0000", 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1571e-05 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.047 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062403+0000", 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011264000000000001 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062751+0000", 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00034876500000000001 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.064547+0000", 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0017959230000000001 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.048 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.066472+0000", 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001924783 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.368547+0000", 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30207485899999997 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.368744+0000", 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019718200000000001 2024-09-16T04:41:54.049 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:137 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.9751710119999997, 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50711364199999998, 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.050 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 137 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.051 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062314+0000", 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062318+0000", 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1710000000000004e-06 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062327+0000", 2024-09-16T04:41:54.052 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0089999999999996e-06 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162444+0000", 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011658 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163173+0000", 2024-09-16T04:41:54.053 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072917800000000003 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163440+0000", 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026697899999999998 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164004+0000", 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056357499999999997 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.054 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164775+0000", 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00077136499999999998 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569249+0000", 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.404473996 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.055 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569427+0000", 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177906 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:138 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.8750021290000003, 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.71018989099999996, 2024-09-16T04:41:54.056 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 138 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.057 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162484+0000", 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162491+0000", 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:41:54.058 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162507+0000", 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5141e-05 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162514+0000", 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8460000000000008e-06 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165370+0000", 2024-09-16T04:41:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028555719999999998 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165438+0000", 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8466999999999993e-05 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570190+0000", 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40475178899999997 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570271+0000", 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0437000000000003e-05 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.872672+0000", 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30240151100000001 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.1s0 36/32 ECSubReadReply(tid=28, attrs_read=0))", 2024-09-16T04:41:54.061 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.162971+0000", 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.8745134859999997, 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20401375799999999, 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162971+0000", 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.062 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162971+0000", 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162971+0000", 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162971+0000", 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163025+0000", 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4564999999999997e-05 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.263188+0000", 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016217500000001 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.366875+0000", 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103687058 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.366985+0000", 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010996 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.3s2 36/32 ECSubWrite(tid=168, reqid=client.4728.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=36'3))", 2024-09-16T04:41:54.065 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.600281+0000", 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 7.4372032780000001, 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20107765699999999, 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.600281+0000", 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.066 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.600281+0000", 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.600283+0000", 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9860000000000001e-06 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.600308+0000", 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4719000000000002e-05 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.600342+0000", 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4622999999999997e-05 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.700476+0000", 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.1001336 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.800656+0000", 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10018036299999999 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.800665+0000", 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2400000000000007e-06 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.801324+0000", 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00065926899999999998 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.801359+0000", 2024-09-16T04:41:54.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4857000000000002e-05 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:181 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9451288629999999, 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.514197601, 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.070 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 181 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092358+0000", 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9500000000000001e-06 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093295+0000", 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00093678900000000005 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093330+0000", 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5039000000000001e-05 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093348+0000", 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7210999999999998e-05 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269509+0000", 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17616195900000001 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269804+0000", 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029419599999999999 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272744+0000", 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029401789999999998 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279866+0000", 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007122499 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606426+0000", 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32655929 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606553+0000", 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000127489 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:182 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9388004179999996, 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50813697599999996, 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.076 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 182 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0260000000000002e-06 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099161+0000", 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000474556 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099192+0000", 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1612000000000002e-05 2024-09-16T04:41:54.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099207+0000", 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4796e-05 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275225+0000", 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17601765799999999 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275471+0000", 2024-09-16T04:41:54.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024600099999999998 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001437422 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280012+0000", 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031043899999999998 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:41:54.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606726+0000", 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32671404399999998 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606821+0000", 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4470999999999997e-05 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.081 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:183 3.bs0 3:d388eed4:::smithi16530038-28:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9382500089999999, 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60643071599999998, 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 183 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.082 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099235+0000", 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.017e-06 2024-09-16T04:41:54.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099242+0000", 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4230000000000002e-06 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099256+0000", 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3936e-05 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099264+0000", 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278076+0000", 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17881208600000001 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278318+0000", 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024238100000000001 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279368+0000", 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010492209999999999 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281553+0000", 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021853739999999999 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705572+0000", 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.42401889100000001 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705665+0000", 2024-09-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3173999999999999e-05 2024-09-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:191 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9363898539999997, 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68460226999999996, 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 191 2024-09-16T04:41:54.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101096+0000", 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-06 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101104+0000", 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1240000000000005e-06 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101122+0000", 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8045000000000001e-05 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101130+0000", 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2670000000000006e-06 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280859+0000", 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179728363 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280864+0000", 2024-09-16T04:41:54.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7370000000000003e-06 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.405994+0000", 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.12512986500000001 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.406243+0000", 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024852700000000001 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.785697+0000", 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.37945393900000002 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:192 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9363376360000002, 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68518017200000003, 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.093 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 192 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101154+0000", 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3300000000000004e-06 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101160+0000", 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.348e-06 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101164+0000", 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2899999999999996e-06 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281071+0000", 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179906749 2024-09-16T04:41:54.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281074+0000", 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.976e-06 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506776+0000", 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22570168299999999 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506893+0000", 2024-09-16T04:41:54.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011752300000000001 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786327+0000", 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27943377000000003 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:199 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:41:54.098 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9362632770000001, 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50411928699999997, 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 199 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:41:54.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101222+0000", 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3799999999999997e-07 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101227+0000", 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8559999999999998e-06 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101240+0000", 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3485e-05 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101248+0000", 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3699999999999995e-06 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280982+0000", 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17973399200000001 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281111+0000", 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012887099999999999 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.605340+0000", 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32422917499999998 2024-09-16T04:41:54.102 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:200 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.936226403, 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50471661899999998, 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.103 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 200 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7300000000000001e-07 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101262+0000", 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.14e-06 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101269+0000", 2024-09-16T04:41:54.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6399999999999997e-06 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101275+0000", 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6910000000000002e-06 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281623+0000", 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18034850899999999 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281792+0000", 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016865700000000001 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.605975+0000", 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32418250900000001 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:201 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:41:54.107 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9362045490000002, 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68324074999999995, 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 201 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:41:54.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101282+0000", 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9750000000000001e-06 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101287+0000", 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.651e-06 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.201419+0000", 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10013264099999999 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482351+0000", 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28093166899999999 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482482+0000", 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013094199999999999 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.784521+0000", 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30203853899999999 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.111 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.1s0 38/37 ECSubReadReply(tid=41, attrs_read=0))", 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.585021+0000", 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.4524633810000003, 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20156737899999999, 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.585021+0000", 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.585021+0000", 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.585022+0000", 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7999999999999995e-07 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.585257+0000", 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00023520099999999999 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.585271+0000", 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3903000000000001e-05 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.685353+0000", 2024-09-16T04:41:54.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100081897 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786540+0000", 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10118719399999999 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786588+0000", 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8603999999999997e-05 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.8.22:2 3.1s0 3:803446a7:::smithi16530038-31:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.411654+0000", 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.625830229, 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.400909505, 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.411654+0000", 2024-09-16T04:41:54.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.411654+0000", 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.411658+0000", 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4429999999999999e-06 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.411670+0000", 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.19e-05 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.411711+0000", 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1232000000000001e-05 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.411734+0000", 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3479e-05 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.509995+0000", 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098260359000000005 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.510119+0000", 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000124702 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.812564+0000", 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30244439000000001 2024-09-16T04:41:54.119 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.1s0 42/37 ECSubReadReply(tid=52, attrs_read=0))", 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.611379+0000", 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.426104985, 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20123126199999999, 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.611379+0000", 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.611379+0000", 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.611381+0000", 2024-09-16T04:41:54.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.373e-06 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.611655+0000", 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027442100000000001 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.611688+0000", 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2462000000000002e-05 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.711829+0000", 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100141458 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.812059+0000", 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100229811 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.812611+0000", 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00055173699999999995 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.123 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.14s2 42/41 ECSubWrite(tid=24, reqid=client.4728.0:198, at_version=42'14, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.623517+0000", 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.4139669429999999, 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20181321899999999, 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.124 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623517+0000", 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623517+0000", 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623519+0000", 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.767e-06 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623591+0000", 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1307e-05 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623620+0000", 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9946000000000001e-05 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.623641+0000", 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0115999999999998e-05 2024-09-16T04:41:54.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.723913+0000", 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100272867 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.723929+0000", 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5285999999999999e-05 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:41:54.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.825296+0000", 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10136706400000001 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.825331+0000", 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4866000000000001e-05 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDPGPushReply(3.17s0 42/41 [PushReplyOp(3:ed82f61a:::smithi16530038-1:head)])", 2024-09-16T04:41:54.128 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.835163+0000", 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2023217820000001, 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20176201599999999, 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.835163+0000", 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.835163+0000", 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.835164+0000", 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5099999999999999e-06 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.835175+0000", 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0837e-05 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.835206+0000", 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1337999999999997e-05 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.835238+0000", 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1612000000000002e-05 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.036835+0000", 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201596689 2024-09-16T04:41:54.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.036925+0000", 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0030000000000004e-05 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDPGPushReply(3.17s0 43/41 [PushReplyOp(3:ed82f61a:::smithi16530038-1:2)])", 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.372603776, 2024-09-16T04:41:54.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20391833100000001, 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664881+0000", 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664919+0000", 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7880999999999999e-05 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.664938+0000", 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9510000000000001e-05 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:41:54.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.865825+0000", 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20088736900000001 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.868799+0000", 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029735709999999999 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.136 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:54.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:41:54.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:105: done (7 left) 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:106: rmattr oid 21 current snap is 3 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:106: done (7 left) 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:107: read oid 29 snap 1 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:107: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:108: rmattr oid 5 current snap is 3 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:108: done (8 left) 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:109: read oid 12 snap 1 2024-09-16T04:41:54.140 INFO:tasks.rados.rados.0.smithi165.stdout:109: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:41:54.141 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 49 current snap is 3 2024-09-16T04:41:54.141 INFO:tasks.rados.rados.0.smithi165.stdout:110: seq_num 59 ranges {819200=622592,1441792=376832} 2024-09-16T04:41:54.141 INFO:tasks.rados.rados.0.smithi165.stdout:110: writing smithi16530038-49 from 819200 to 1441792 tid 1 2024-09-16T04:41:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:41:54.142 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:54.132+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:110: writing smithi16530038-49 from 1441792 to 1818624 tid 2 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:111: copy_from oid 39 from oid 14 current snap is 3 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:75: finishing copy_from to smithi16530038-1 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 1 to smithi16530038-19 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:58: finishing copy_from to smithi16530038-30 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T04:41:54.143 INFO:tasks.rados.rados.0.smithi165.stdout:102: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T04:41:54.186 INFO:tasks.rados.rados.0.smithi165.stdout:76: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:103: finishing copy_from racing read to smithi16530038-42 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 2 to smithi16530038-19 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 3 to smithi16530038-19 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:103: finishing copy_from to smithi16530038-42 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 3 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 4 to smithi16530038-19 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:93: oid 19 updating version 0 to 7 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:93: oid 19 version 7 is already newer than 6 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:93: oid 19 updating version 7 to 8 2024-09-16T04:41:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 8 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:41:54.191 INFO:tasks.rados.rados.0.smithi165.stdout:93: left oid 19 (ObjNum 57 snap 2 seq_num 57) 2024-09-16T04:41:54.191 INFO:tasks.rados.rados.0.smithi165.stdout:101: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:41:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:58: done (10 left) 2024-09-16T04:41:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:75: done (9 left) 2024-09-16T04:41:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:76: done (8 left) 2024-09-16T04:41:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:93: done (7 left) 2024-09-16T04:41:54.197 INFO:tasks.rados.rados.0.smithi165.stdout:101: done (6 left) 2024-09-16T04:41:54.197 INFO:tasks.rados.rados.0.smithi165.stdout:102: done (5 left) 2024-09-16T04:41:54.197 INFO:tasks.rados.rados.0.smithi165.stdout:103: done (4 left) 2024-09-16T04:41:54.197 INFO:tasks.rados.rados.0.smithi165.stdout:112: copy_from oid 2 from oid 26 current snap is 3 2024-09-16T04:41:54.197 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 46 current snap is 3 2024-09-16T04:41:54.197 INFO:tasks.rados.rados.0.smithi165.stdout:113: seq_num 60 ranges {770048=638976,1409024=376832} 2024-09-16T04:41:54.220 INFO:tasks.rados.rados.0.smithi165.stdout:113: writing smithi16530038-46 from 770048 to 1409024 tid 1 2024-09-16T04:41:54.227 INFO:tasks.rados.rados.0.smithi165.stdout:113: writing smithi16530038-46 from 1409024 to 1785856 tid 2 2024-09-16T04:41:54.227 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 47 current snap is 3 2024-09-16T04:41:54.227 INFO:tasks.rados.rados.0.smithi165.stdout:114: seq_num 61 ranges {786432=655360,1441792=376832} 2024-09-16T04:41:54.239 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:54.236+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:54.248 INFO:tasks.rados.rados.0.smithi165.stdout:114: writing smithi16530038-47 from 786432 to 1441792 tid 1 2024-09-16T04:41:54.253 INFO:tasks.rados.rados.0.smithi165.stdout:114: writing smithi16530038-47 from 1441792 to 1818624 tid 2 2024-09-16T04:41:54.253 INFO:tasks.rados.rados.0.smithi165.stdout:115: rollback oid 27 current snap is 3 2024-09-16T04:41:54.253 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 27 to 1 2024-09-16T04:41:54.253 INFO:tasks.rados.rados.0.smithi165.stdout:116: delete oid 31 current snap is 3 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:294 3.14s0 3.29d4b2d4 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.144405+0000", 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.17245468, 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17248171400000001, 2024-09-16T04:41:54.317 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 294 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144405+0000", 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144405+0000", 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144405+0000", 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9299999999999999e-07 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144407+0000", 2024-09-16T04:41:54.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4810000000000001e-06 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144412+0000", 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5400000000000003e-06 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144416+0000", 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1590000000000003e-06 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.320 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 44/41 ECSubWriteReply(tid=31, last_complete=44'17, committed=1, applied=1))", 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.213926+0000", 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.102933335, 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103032756, 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.213926+0000", 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.213926+0000", 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.213927+0000", 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5199999999999997e-07 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.213930+0000", 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6229999999999999e-06 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.213940+0000", 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6169999999999996e-06 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.213947+0000", 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2030000000000003e-06 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.323 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 43/41 ECSubWriteReply(tid=31, last_complete=44'17, committed=1, applied=1))", 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.117620+0000", 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.19923990899999999, 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19939126200000001, 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117620+0000", 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117620+0000", 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117621+0000", 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.418e-06 2024-09-16T04:41:54.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117633+0000", 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2187999999999999e-05 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117652+0000", 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9184000000000001e-05 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117663+0000", 2024-09-16T04:41:54.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=56, reqid=osd.11.0:56, at_version=44'19, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.19796488000000001, 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.198166381, 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.327 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6400000000000003e-07 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118922+0000", 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6635000000000002e-05 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118941+0000", 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9707999999999998e-05 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118952+0000", 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.039e-05 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=59, reqid=osd.11.0:59, at_version=44'21, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.19509943699999999, 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19534933400000001, 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.330 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121761+0000", 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0699999999999996e-07 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121787+0000", 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6227000000000001e-05 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121804+0000", 2024-09-16T04:41:54.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6898999999999999e-05 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121815+0000", 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1294000000000001e-05 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:291 3.14s0 3.29d4b2d4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.137888+0000", 2024-09-16T04:41:54.333 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.17897196600000001, 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17926309600000001, 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 291 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.137888+0000", 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.137888+0000", 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.137890+0000", 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.605e-06 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.138743+0000", 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00085253399999999995 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.138765+0000", 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2558e-05 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.138777+0000", 2024-09-16T04:41:54.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1498000000000001e-05 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:292 3.14s0 3.29d4b2d4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.143912+0000", 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.172947353, 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17327216300000001, 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.337 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 292 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.143912+0000", 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.143912+0000", 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.143913+0000", 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5099999999999998e-07 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144359+0000", 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00044600500000000002 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144370+0000", 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1063000000000001e-05 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144376+0000", 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8969999999999996e-06 2024-09-16T04:41:54.340 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:293 3.14s0 3.29d4b2d4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.144386+0000", 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.172473878, 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.172855814, 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:41:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 293 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144386+0000", 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144386+0000", 2024-09-16T04:41:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144386+0000", 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4400000000000001e-07 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144392+0000", 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5350000000000004e-06 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144397+0000", 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0549999999999999e-06 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.144400+0000", 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.337e-06 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 8 2024-09-16T04:41:54.344 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:54.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:41:54.346 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:54.347 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:54.347 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:54.344+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:54.361 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:54.361 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:54.382 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:54.382 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:54.402 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:54.403 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:54.416 INFO:tasks.rados.rados.0.smithi165.stdout:109: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:41:54.446 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:41:54.440+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:41:54.482 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:54.483 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:54.483 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:41:54.483 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:41:54.483 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:54.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:41:54.523 INFO:tasks.rados.rados.0.smithi165.stdout:107: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:41:54.540 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:54.567 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:54.567 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:54.567 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:54.564+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:54.607 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:54.608 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:54.634 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:54.634 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:54.648 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:54.644+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:54.656 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:54.657 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:54.677 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:54.678 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:54.678 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:54.678 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:54.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:41:54.716 INFO:tasks.rados.rados.0.smithi165.stdout:111: finishing copy_from to smithi16530038-39 2024-09-16T04:41:54.716 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T04:41:54.720 INFO:tasks.rados.rados.0.smithi165.stdout:110: finishing write tid 1 to smithi16530038-49 2024-09-16T04:41:54.749 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:54.744+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:54.831 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 1 to smithi16530038-46 2024-09-16T04:41:54.831 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 2 to smithi16530038-46 2024-09-16T04:41:54.831 INFO:tasks.rados.rados.0.smithi165.stdout:111: finishing copy_from racing read to smithi16530038-39 2024-09-16T04:41:54.832 INFO:tasks.rados.rados.0.smithi165.stdout:110: finishing write tid 2 to smithi16530038-49 2024-09-16T04:41:54.832 INFO:tasks.rados.rados.0.smithi165.stdout:110: finishing write tid 3 to smithi16530038-49 2024-09-16T04:41:54.835 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:54.835 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:54.850 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:54.848+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:54.865 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:41:54.865 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:41:54.865 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:41:54.865 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:41:54.879 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:41:54.951 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:41:54.947+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:41:54.954 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:54.955 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:54.974 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:54.975 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:54.998 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:54.998 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:55.015 INFO:tasks.rados.rados.0.smithi165.stdout:112: finishing copy_from racing read to smithi16530038-2 2024-09-16T04:41:55.034 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 3 to smithi16530038-46 2024-09-16T04:41:55.041 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:55.051 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:55.051 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:55.051 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:55.048+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:55.063 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:55.063 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:55.065 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:41:55.102 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:41:55.102 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:41:55.103 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:41:55.130 INFO:tasks.rados.rados.0.smithi165.stdout:114: finishing write tid 1 to smithi16530038-47 2024-09-16T04:41:55.130 INFO:tasks.rados.rados.0.smithi165.stdout:110: finishing write tid 4 to smithi16530038-49 2024-09-16T04:41:55.130 INFO:tasks.rados.rados.0.smithi165.stdout:110: oid 49 updating version 0 to 19 2024-09-16T04:41:55.130 INFO:tasks.rados.rados.0.smithi165.stdout:110: oid 49 updating version 19 to 21 2024-09-16T04:41:55.131 INFO:tasks.rados.rados.0.smithi165.stdout:110: oid 49 version 21 is already newer than 20 2024-09-16T04:41:55.131 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 21 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-09-16T04:41:55.131 INFO:tasks.rados.rados.0.smithi165.stdout:110: left oid 49 (ObjNum 59 snap 3 seq_num 59) 2024-09-16T04:41:55.152 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:55.147+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:55.222 INFO:tasks.rados.rados.0.smithi165.stdout:112: finishing copy_from to smithi16530038-2 2024-09-16T04:41:55.223 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:41:55.231 INFO:tasks.rados.rados.0.smithi165.stdout:114: finishing write tid 2 to smithi16530038-47 2024-09-16T04:41:55.231 INFO:tasks.rados.rados.0.smithi165.stdout:114: finishing write tid 3 to smithi16530038-47 2024-09-16T04:41:55.232 INFO:tasks.rados.rados.0.smithi165.stdout:115: finishing rollback tid 0 to smithi16530038-27 2024-09-16T04:41:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:107: done (8 left) 2024-09-16T04:41:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:109: done (7 left) 2024-09-16T04:41:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:110: done (6 left) 2024-09-16T04:41:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:111: done (5 left) 2024-09-16T04:41:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:112: done (4 left) 2024-09-16T04:41:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:116: done (3 left) 2024-09-16T04:41:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:117: read oid 20 snap -1 2024-09-16T04:41:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:117: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T04:41:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:118: rollback oid 42 current snap is 3 2024-09-16T04:41:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 1 2024-09-16T04:41:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:119: delete oid 15 current snap is 3 2024-09-16T04:41:55.253 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:55.252+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:55.260 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:55.261 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:55.276 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:41:55.340 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 4 to smithi16530038-46 2024-09-16T04:41:55.340 INFO:tasks.rados.rados.0.smithi165.stdout:113: oid 46 updating version 0 to 18 2024-09-16T04:41:55.340 INFO:tasks.rados.rados.0.smithi165.stdout:113: oid 46 updating version 18 to 19 2024-09-16T04:41:55.340 INFO:tasks.rados.rados.0.smithi165.stdout:113: oid 46 updating version 19 to 20 2024-09-16T04:41:55.340 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 20 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-16T04:41:55.340 INFO:tasks.rados.rados.0.smithi165.stdout:113: left oid 46 (ObjNum 60 snap 3 seq_num 60) 2024-09-16T04:41:55.354 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:55.352+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:55.407 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:55.407 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:55.421 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:41:55.438 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:41:55.438 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:41:55.456 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-09-16T04:41:55.116493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30048819,"num_objects":69,"num_object_clones":17,"num_object_copies":463,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":293,"num_read_kb":12897,"num_write":359,"num_write_kb":31238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":193,"acting":193,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":221,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":501988,"kb_used_data":57036,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509447452,"statfs":{"total":1546188226560,"available":1545674190848,"internally_reserved":0,"allocated":58404864,"data_stored":67977225,"data_compressed":1012007,"data_compressed_allocated":1425408,"data_compressed_original":3710976,"omap_allocated":25522,"internal_metadata":455253070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,6,1,6,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10027008,"num_objects":18,"num_object_clones":12,"num_object_copies":126,"num_objects_missing_on_primary":-2,"num_objects_missing":-3,"num_objects_degraded":-14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":49,"num_read_kb":4928,"num_write":57,"num_write_kb":9088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026621"},"pg_stats":[{"pgid":"3.19","version":"44'19","reported_seq":85,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:54.117379+0000","last_change":"2024-09-16T04:41:54.117379+0000","last_active":"2024-09-16T04:41:54.117379+0000","last_peered":"2024-09-16T04:41:54.117379+0000","last_clean":"2024-09-16T04:41:54.117379+0000","last_became_active":"2024-09-16T04:41:52.308003+0000","last_became_peered":"2024-09-16T04:41:52.308003+0000","last_unstale":"2024-09-16T04:41:54.117379+0000","last_undegraded":"2024-09-16T04:41:54.117379+0000","last_fullsized":"2024-09-16T04:41:54.117379+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001394162,"stat_sum":{"num_bytes":2162688,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":196,"num_write":10,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.18","version":"35'2","reported_seq":29,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.114981+0000","last_change":"2024-09-16T04:41:54.114981+0000","last_active":"2024-09-16T04:41:54.114981+0000","last_peered":"2024-09-16T04:41:54.114981+0000","last_clean":"2024-09-16T04:41:54.114981+0000","last_became_active":"2024-09-16T04:41:52.415756+0000","last_became_peered":"2024-09-16T04:41:52.415756+0000","last_unstale":"2024-09-16T04:41:54.114981+0000","last_undegraded":"2024-09-16T04:41:54.114981+0000","last_fullsized":"2024-09-16T04:41:54.114981+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027803749999999999,"stat_sum":{"num_bytes":172032,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.17","version":"40'4","reported_seq":85,"reported_epoch":44,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-16T04:41:54.218204+0000","last_change":"2024-09-16T04:41:54.218132+0000","last_active":"2024-09-16T04:41:54.218204+0000","last_peered":"2024-09-16T04:41:54.218204+0000","last_clean":"2024-09-16T04:41:54.218204+0000","last_became_active":"2024-09-16T04:41:52.612293+0000","last_became_peered":"2024-09-16T04:41:52.612293+0000","last_unstale":"2024-09-16T04:41:54.218204+0000","last_undegraded":"2024-09-16T04:41:54.218204+0000","last_fullsized":"2024-09-16T04:41:54.218204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001369632,"stat_sum":{"num_bytes":622592,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.16","version":"35'2","reported_seq":45,"reported_epoch":42,"state":"active","last_fresh":"2024-09-16T04:41:52.305704+0000","last_change":"2024-09-16T04:41:52.305704+0000","last_active":"2024-09-16T04:41:52.305704+0000","last_peered":"2024-09-16T04:41:52.305704+0000","last_clean":"2024-09-16T04:41:44.952133+0000","last_became_active":"2024-09-16T04:41:52.305704+0000","last_became_peered":"2024-09-16T04:41:52.305704+0000","last_unstale":"2024-09-16T04:41:52.305704+0000","last_undegraded":"2024-09-16T04:41:52.305704+0000","last_fullsized":"2024-09-16T04:41:52.305704+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40960,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":["0(0)","1(2)","4(2)","5(5)","7(5)","8(3)","9(4)","9(6)","10(6)","11(0)","12(4)","13(1)","15(1)","15(3)"],"object_location_counts":[{"shards":"0(0),1(2),7(5),9(6),12(4),13(1),15(3)","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"36'6","reported_seq":21,"reported_epoch":44,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T04:41:54.125439+0000","last_change":"2024-09-16T04:41:53.449148+0000","last_active":"2024-09-16T04:41:54.125439+0000","last_peered":"2024-09-16T04:41:54.125439+0000","last_clean":"2024-09-16T04:41:46.181856+0000","last_became_active":"2024-09-16T04:41:53.445446+0000","last_became_peered":"2024-09-16T04:41:53.445446+0000","last_unstale":"2024-09-16T04:41:54.125439+0000","last_undegraded":"2024-09-16T04:41:54.125439+0000","last_fullsized":"2024-09-16T04:41:54.125439+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":761856,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,14,0,6],"avail_no_missing":["15(0)","0(5)","2(5)","3(2)","4(2)","6(1)","6(6)","7(4)","8(1)","9(0)","13(3)","14(4)","15(6)"],"object_location_counts":[{"shards":"0(5),4(2),6(6),8(1),13(3),14(4),15(0)","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.14","version":"44'19","reported_seq":81,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:54.512583+0000","last_change":"2024-09-16T04:41:54.111682+0000","last_active":"2024-09-16T04:41:54.512583+0000","last_peered":"2024-09-16T04:41:54.512583+0000","last_clean":"2024-09-16T04:41:54.512583+0000","last_became_active":"2024-09-16T04:41:52.409152+0000","last_became_peered":"2024-09-16T04:41:52.409152+0000","last_unstale":"2024-09-16T04:41:54.512583+0000","last_undegraded":"2024-09-16T04:41:54.512583+0000","last_fullsized":"2024-09-16T04:41:54.512583+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.202002812,"stat_sum":{"num_bytes":1867776,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":147,"num_write":9,"num_write_kb":2064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.123998+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:54.123998+0000","last_peered":"2024-09-16T04:41:54.123998+0000","last_clean":"2024-09-16T04:41:54.123998+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:54.123998+0000","last_undegraded":"2024-09-16T04:41:54.123998+0000","last_fullsized":"2024-09-16T04:41:54.123998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":57,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:41:52.098865+0000","last_change":"2024-09-16T04:41:50.605722+0000","last_active":"2024-09-16T04:41:52.098865+0000","last_peered":"2024-09-16T04:41:52.098865+0000","last_clean":"2024-09-16T04:41:52.098865+0000","last_became_active":"2024-09-16T04:41:48.483004+0000","last_became_peered":"2024-09-16T04:41:48.483004+0000","last_unstale":"2024-09-16T04:41:52.098865+0000","last_undegraded":"2024-09-16T04:41:52.098865+0000","last_fullsized":"2024-09-16T04:41:52.098865+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:06.776639+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.111485+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:54.111485+0000","last_peered":"2024-09-16T04:41:54.111485+0000","last_clean":"2024-09-16T04:41:54.111485+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:54.111485+0000","last_undegraded":"2024-09-16T04:41:54.111485+0000","last_fullsized":"2024-09-16T04:41:54.111485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":66,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.118932+0000","last_change":"2024-09-16T04:41:54.119687+0000","last_active":"2024-09-16T04:41:54.118932+0000","last_peered":"2024-09-16T04:41:54.118932+0000","last_clean":"2024-09-16T04:41:54.118932+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:41:54.118932+0000","last_undegraded":"2024-09-16T04:41:54.118932+0000","last_fullsized":"2024-09-16T04:41:54.118932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010071640000000001,"stat_sum":{"num_bytes":16384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.124009+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:54.124009+0000","last_peered":"2024-09-16T04:41:54.124009+0000","last_clean":"2024-09-16T04:41:54.124009+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:54.124009+0000","last_undegraded":"2024-09-16T04:41:54.124009+0000","last_fullsized":"2024-09-16T04:41:54.124009+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"36'10","reported_seq":84,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.113606+0000","last_change":"2024-09-16T04:41:54.214663+0000","last_active":"2024-09-16T04:41:54.113606+0000","last_peered":"2024-09-16T04:41:54.113606+0000","last_clean":"2024-09-16T04:41:54.113606+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:41:54.113606+0000","last_undegraded":"2024-09-16T04:41:54.113606+0000","last_fullsized":"2024-09-16T04:41:54.113606+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.102783712,"stat_sum":{"num_bytes":737280,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":50,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.117713+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:54.117713+0000","last_peered":"2024-09-16T04:41:54.117713+0000","last_clean":"2024-09-16T04:41:54.117713+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:54.117713+0000","last_undegraded":"2024-09-16T04:41:54.117713+0000","last_fullsized":"2024-09-16T04:41:54.117713+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"44'12","reported_seq":80,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:54.127009+0000","last_change":"2024-09-16T04:41:54.127009+0000","last_active":"2024-09-16T04:41:54.127009+0000","last_peered":"2024-09-16T04:41:54.127009+0000","last_clean":"2024-09-16T04:41:54.127009+0000","last_became_active":"2024-09-16T04:41:48.475648+0000","last_became_peered":"2024-09-16T04:41:48.475648+0000","last_unstale":"2024-09-16T04:41:54.127009+0000","last_undegraded":"2024-09-16T04:41:54.127009+0000","last_fullsized":"2024-09-16T04:41:54.127009+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:41:59.265334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101128806,"stat_sum":{"num_bytes":1744896,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.3","version":"36'3","reported_seq":55,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T04:41:50.084226+0000","last_change":"2024-09-16T04:41:49.294305+0000","last_active":"2024-09-16T04:41:50.084226+0000","last_peered":"2024-09-16T04:41:50.084226+0000","last_clean":"2024-09-16T04:41:50.084226+0000","last_became_active":"2024-09-16T04:41:48.281236+0000","last_became_peered":"2024-09-16T04:41:48.281236+0000","last_unstale":"2024-09-16T04:41:50.084226+0000","last_undegraded":"2024-09-16T04:41:50.084226+0000","last_fullsized":"2024-09-16T04:41:50.084226+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:41:49.293927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.293927+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:10.195310+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":835584,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":53,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.208275+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:54.208275+0000","last_peered":"2024-09-16T04:41:54.208275+0000","last_clean":"2024-09-16T04:41:54.208275+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:54.208275+0000","last_undegraded":"2024-09-16T04:41:54.208275+0000","last_fullsized":"2024-09-16T04:41:54.208275+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":73,"reported_epoch":44,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-16T04:41:54.117712+0000","last_change":"2024-09-16T04:41:54.117654+0000","last_active":"2024-09-16T04:41:54.117712+0000","last_peered":"2024-09-16T04:41:54.117712+0000","last_clean":"2024-09-16T04:41:54.117712+0000","last_became_active":"2024-09-16T04:41:48.383143+0000","last_became_peered":"2024-09-16T04:41:48.383143+0000","last_unstale":"2024-09-16T04:41:54.117712+0000","last_undegraded":"2024-09-16T04:41:54.117712+0000","last_fullsized":"2024-09-16T04:41:54.117712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:49.599894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.599894+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:14.775582+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0.0024063940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T04:41:50.083445+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:41:50.083445+0000","last_peered":"2024-09-16T04:41:50.083445+0000","last_clean":"2024-09-16T04:41:50.083445+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:41:50.083445+0000","last_undegraded":"2024-09-16T04:41:50.083445+0000","last_fullsized":"2024-09-16T04:41:50.083445+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"36'10","reported_seq":90,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.117789+0000","last_change":"2024-09-16T04:41:54.118356+0000","last_active":"2024-09-16T04:41:54.117789+0000","last_peered":"2024-09-16T04:41:54.117789+0000","last_clean":"2024-09-16T04:41:54.117789+0000","last_became_active":"2024-09-16T04:41:48.404175+0000","last_became_peered":"2024-09-16T04:41:48.404175+0000","last_unstale":"2024-09-16T04:41:54.117789+0000","last_undegraded":"2024-09-16T04:41:54.117789+0000","last_fullsized":"2024-09-16T04:41:54.117789+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:50.183380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.183380+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:58.674771+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0.00085395200000000005,"stat_sum":{"num_bytes":1146880,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.207723+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:54.207723+0000","last_peered":"2024-09-16T04:41:54.207723+0000","last_clean":"2024-09-16T04:41:54.207723+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:54.207723+0000","last_undegraded":"2024-09-16T04:41:54.207723+0000","last_fullsized":"2024-09-16T04:41:54.207723+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"36'4","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T04:41:50.564685+0000","last_change":"2024-09-16T04:41:50.564791+0000","last_active":"2024-09-16T04:41:50.564685+0000","last_peered":"2024-09-16T04:41:50.564685+0000","last_clean":"2024-09-16T04:41:50.564685+0000","last_became_active":"2024-09-16T04:41:48.477867+0000","last_became_peered":"2024-09-16T04:41:48.477867+0000","last_unstale":"2024-09-16T04:41:50.564685+0000","last_undegraded":"2024-09-16T04:41:50.564685+0000","last_fullsized":"2024-09-16T04:41:50.564685+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.564481+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:09.149760+0000","scrub_duration":1083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":52,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.112248+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:54.112248+0000","last_peered":"2024-09-16T04:41:54.112248+0000","last_clean":"2024-09-16T04:41:54.112248+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:54.112248+0000","last_undegraded":"2024-09-16T04:41:54.112248+0000","last_fullsized":"2024-09-16T04:41:54.112248+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":237,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T04:41:50.278456+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:41:50.278456+0000","last_peered":"2024-09-16T04:41:50.278456+0000","last_clean":"2024-09-16T04:41:50.278456+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:41:50.278456+0000","last_undegraded":"2024-09-16T04:41:50.278456+0000","last_fullsized":"2024-09-16T04:41:50.278456+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"44'8","reported_seq":80,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.421631+0000","last_change":"2024-09-16T04:41:54.423388+0000","last_active":"2024-09-16T04:41:54.421631+0000","last_peered":"2024-09-16T04:41:54.421631+0000","last_clean":"2024-09-16T04:41:54.421631+0000","last_became_active":"2024-09-16T04:41:48.481646+0000","last_became_peered":"2024-09-16T04:41:48.481646+0000","last_unstale":"2024-09-16T04:41:54.421631+0000","last_undegraded":"2024-09-16T04:41:54.421631+0000","last_fullsized":"2024-09-16T04:41:54.421631+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'4","last_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.115829+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:09.765928+0000","scrub_duration":1211,"objects_trimmed":1,"snaptrim_duration":0.31032031700000001,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.9","version":"36'11","reported_seq":85,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.117472+0000","last_change":"2024-09-16T04:41:54.118643+0000","last_active":"2024-09-16T04:41:54.117472+0000","last_peered":"2024-09-16T04:41:54.117472+0000","last_clean":"2024-09-16T04:41:54.117472+0000","last_became_active":"2024-09-16T04:41:48.676866+0000","last_became_peered":"2024-09-16T04:41:48.676866+0000","last_unstale":"2024-09-16T04:41:54.117472+0000","last_undegraded":"2024-09-16T04:41:54.117472+0000","last_fullsized":"2024-09-16T04:41:54.117472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'11","last_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:41:51.621244+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:14.135321+0000","scrub_duration":1979,"objects_trimmed":0,"snaptrim_duration":0.001844774,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.a","version":"38'12","reported_seq":86,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:54.309159+0000","last_change":"2024-09-16T04:41:54.309229+0000","last_active":"2024-09-16T04:41:54.309159+0000","last_peered":"2024-09-16T04:41:54.309159+0000","last_clean":"2024-09-16T04:41:54.309159+0000","last_became_active":"2024-09-16T04:41:42.059056+0000","last_became_peered":"2024-09-16T04:41:42.059056+0000","last_unstale":"2024-09-16T04:41:54.309159+0000","last_undegraded":"2024-09-16T04:41:54.309159+0000","last_fullsized":"2024-09-16T04:41:54.309159+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:08.496013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40364518799999999,"stat_sum":{"num_bytes":1769472,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":3273,"num_write":11,"num_write_kb":2400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.b","version":"38'10","reported_seq":73,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:54.416132+0000","last_change":"2024-09-16T04:41:54.215928+0000","last_active":"2024-09-16T04:41:54.416132+0000","last_peered":"2024-09-16T04:41:54.416132+0000","last_clean":"2024-09-16T04:41:54.416132+0000","last_became_active":"2024-09-16T04:41:42.059548+0000","last_became_peered":"2024-09-16T04:41:42.059548+0000","last_unstale":"2024-09-16T04:41:54.416132+0000","last_undegraded":"2024-09-16T04:41:54.416132+0000","last_fullsized":"2024-09-16T04:41:54.416132+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:57.005265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010531959999999999,"stat_sum":{"num_bytes":1818624,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":1927,"num_write":9,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.c","version":"39'17","reported_seq":63,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:41:52.101906+0000","last_change":"2024-09-16T04:41:42.059491+0000","last_active":"2024-09-16T04:41:52.101906+0000","last_peered":"2024-09-16T04:41:52.101906+0000","last_clean":"2024-09-16T04:41:52.101906+0000","last_became_active":"2024-09-16T04:41:42.057674+0000","last_became_peered":"2024-09-16T04:41:42.057674+0000","last_unstale":"2024-09-16T04:41:52.101906+0000","last_undegraded":"2024-09-16T04:41:52.101906+0000","last_fullsized":"2024-09-16T04:41:52.101906+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:47.459772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":1543,"num_write":34,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"43'36","reported_seq":97,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T04:41:53.618620+0000","last_change":"2024-09-16T04:41:53.404326+0000","last_active":"2024-09-16T04:41:53.618620+0000","last_peered":"2024-09-16T04:41:53.618620+0000","last_clean":"2024-09-16T04:41:53.618620+0000","last_became_active":"2024-09-16T04:41:42.053696+0000","last_became_peered":"2024-09-16T04:41:42.053696+0000","last_unstale":"2024-09-16T04:41:53.618620+0000","last_undegraded":"2024-09-16T04:41:53.618620+0000","last_fullsized":"2024-09-16T04:41:53.618620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:43.101130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.201996394,"stat_sum":{"num_bytes":5668864,"num_objects":12,"num_object_clones":5,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":25,"num_read_kb":2795,"num_write":28,"num_write_kb":5536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.e","version":"36'8","reported_seq":71,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T04:41:54.113625+0000","last_change":"2024-09-16T04:41:54.214433+0000","last_active":"2024-09-16T04:41:54.113625+0000","last_peered":"2024-09-16T04:41:54.113625+0000","last_clean":"2024-09-16T04:41:54.113625+0000","last_became_active":"2024-09-16T04:41:42.158174+0000","last_became_peered":"2024-09-16T04:41:42.158174+0000","last_unstale":"2024-09-16T04:41:54.113625+0000","last_undegraded":"2024-09-16T04:41:54.113625+0000","last_fullsized":"2024-09-16T04:41:54.113625+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:53.479303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101302076,"stat_sum":{"num_bytes":1212416,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":2358,"num_write":8,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.f","version":"39'13","reported_seq":57,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T04:41:52.104831+0000","last_change":"2024-09-16T04:41:42.153067+0000","last_active":"2024-09-16T04:41:52.104831+0000","last_peered":"2024-09-16T04:41:52.104831+0000","last_clean":"2024-09-16T04:41:52.104831+0000","last_became_active":"2024-09-16T04:41:41.950892+0000","last_became_peered":"2024-09-16T04:41:41.950892+0000","last_unstale":"2024-09-16T04:41:52.104831+0000","last_undegraded":"2024-09-16T04:41:52.104831+0000","last_fullsized":"2024-09-16T04:41:52.104831+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.197463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1261568,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":532,"num_write":10,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"44'10","reported_seq":69,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:54.118932+0000","last_change":"2024-09-16T04:41:54.118932+0000","last_active":"2024-09-16T04:41:54.118932+0000","last_peered":"2024-09-16T04:41:54.118932+0000","last_clean":"2024-09-16T04:41:54.118932+0000","last_became_active":"2024-09-16T04:41:52.406889+0000","last_became_peered":"2024-09-16T04:41:52.406889+0000","last_unstale":"2024-09-16T04:41:54.118932+0000","last_undegraded":"2024-09-16T04:41:54.118932+0000","last_fullsized":"2024-09-16T04:41:54.118932+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.002144857,"stat_sum":{"num_bytes":655360,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.11","version":"44'19","reported_seq":83,"reported_epoch":44,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:54.117575+0000","last_change":"2024-09-16T04:41:54.117575+0000","last_active":"2024-09-16T04:41:54.117575+0000","last_peered":"2024-09-16T04:41:54.117575+0000","last_clean":"2024-09-16T04:41:54.117575+0000","last_became_active":"2024-09-16T04:41:52.407015+0000","last_became_peered":"2024-09-16T04:41:52.407015+0000","last_unstale":"2024-09-16T04:41:54.117575+0000","last_undegraded":"2024-09-16T04:41:54.117575+0000","last_fullsized":"2024-09-16T04:41:54.117575+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.00069660200000000005,"stat_sum":{"num_bytes":2351104,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.12","version":"36'4","reported_seq":9,"reported_epoch":40,"state":"activating+degraded","last_fresh":"2024-09-16T04:41:50.426623+0000","last_change":"2024-09-16T04:41:50.096016+0000","last_active":"2024-09-16T04:41:50.090802+0000","last_peered":"2024-09-16T04:41:46.066276+0000","last_clean":"2024-09-16T04:41:46.066276+0000","last_became_active":"2024-09-16T04:41:41.957014+0000","last_became_peered":"2024-09-16T04:41:41.957014+0000","last_unstale":"2024-09-16T04:41:50.426623+0000","last_undegraded":"2024-09-16T04:41:50.092108+0000","last_fullsized":"2024-09-16T04:41:50.426623+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516096,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":["1(5)","2(1)","3(2)","6(3)","11(6)","13(0)","15(4)"],"object_location_counts":[{"shards":"1(5),2(1),3(2),6(3),11(6),13(0)","objects":1}],"blocked_by":[1,6],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:41:47.046130+0000","last_change":"2024-09-16T04:41:47.046130+0000","last_active":"2024-09-16T04:41:47.046130+0000","last_peered":"2024-09-16T04:41:47.046130+0000","last_clean":"2024-09-16T04:41:47.046130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:41:47.046130+0000","last_undegraded":"2024-09-16T04:41:47.046130+0000","last_fullsized":"2024-09-16T04:41:47.046130+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:47.046130+0000","last_clean_scrub_stamp":"2024-09-16T04:41:47.046130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":29458432,"num_objects":65,"num_object_clones":17,"num_object_copies":455,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":157,"num_read_kb":12779,"num_write":154,"num_write_kb":27776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":52133888,"data_stored":63852544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":175,"acting":175,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33828,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338012,"statfs":{"total":96636764160,"available":96602124288,"internally_reserved":0,"allocated":2703360,"data_stored":3125540,"data_compressed":83764,"data_compressed_allocated":114688,"data_compressed_original":659456,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29300,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342540,"statfs":{"total":96636764160,"available":96606760960,"internally_reserved":0,"allocated":2318336,"data_stored":2342660,"data_compressed":51819,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30260,"kb_used_data":3232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341580,"statfs":{"total":96636764160,"available":96605777920,"internally_reserved":0,"allocated":3309568,"data_stored":3243780,"data_compressed":51819,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313225,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35908,"kb_used_data":4716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335932,"statfs":{"total":96636764160,"available":96599994368,"internally_reserved":0,"allocated":4829184,"data_stored":5658345,"data_compressed":54760,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313225,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30192,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341648,"statfs":{"total":96636764160,"available":96605847552,"internally_reserved":0,"allocated":3235840,"data_stored":3325800,"data_compressed":57737,"data_compressed_allocated":81920,"data_compressed_original":163840,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280522,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30628,"kb_used_data":3592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341212,"statfs":{"total":96636764160,"available":96605401088,"internally_reserved":0,"allocated":3678208,"data_stored":4483578,"data_compressed":92686,"data_compressed_allocated":126976,"data_compressed_original":684032,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30092,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341748,"statfs":{"total":96636764160,"available":96605949952,"internally_reserved":0,"allocated":3137536,"data_stored":3485780,"data_compressed":63744,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280522,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30720,"kb_used_data":3684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341120,"statfs":{"total":96636764160,"available":96605306880,"internally_reserved":0,"allocated":3772416,"data_stored":4413421,"data_compressed":60741,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,1,6,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280522,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36112,"kb_used_data":4924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335728,"statfs":{"total":96636764160,"available":96599785472,"internally_reserved":0,"allocated":5042176,"data_stored":6240218,"data_compressed":60741,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280522,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29712,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342128,"statfs":{"total":96636764160,"available":96606339072,"internally_reserved":0,"allocated":2752512,"data_stored":2979802,"data_compressed":60741,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280521,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31144,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340696,"statfs":{"total":96636764160,"available":96604872704,"internally_reserved":0,"allocated":4214784,"data_stored":4698196,"data_compressed":63744,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30948,"kb_used_data":3920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340892,"statfs":{"total":96636764160,"available":96605073408,"internally_reserved":0,"allocated":4014080,"data_stored":4732890,"data_compressed":60741,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":4832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":4947968,"data_stored":6137818,"data_compressed":60741,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31172,"kb_used_data":4144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340668,"statfs":{"total":96636764160,"available":96604844032,"internally_reserved":0,"allocated":4243456,"data_stored":5435476,"data_compressed":63744,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28828,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343012,"statfs":{"total":96636764160,"available":96607244288,"internally_reserved":0,"allocated":1843200,"data_stored":1904743,"data_compressed":63744,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31288,"kb_used_data":4260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340552,"statfs":{"total":96636764160,"available":96604725248,"internally_reserved":0,"allocated":4362240,"data_stored":5769178,"data_compressed":60741,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4489216,"data_stored":5500928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2887680,"data_stored":3002368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3461120,"data_stored":4132864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3211264,"data_stored":3964928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4825088,"data_stored":6483968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2519040,"data_stored":3145728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3743744,"data_stored":4415488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3170304,"data_stored":3997696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":6045696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1896448,"data_stored":2101248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3866624,"data_stored":5246976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2789376,"data_stored":3063808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1368064,"data_stored":1622016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":2371584,"data_stored":2662400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4075520,"data_stored":5263360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":2666496,"data_stored":3203072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:41:55.458 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.12 2024-09-16T04:41:55.467 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:55.464+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:55.528 INFO:tasks.rados.rados.0.smithi165.stdout:113: done (5 left) 2024-09-16T04:41:55.528 INFO:tasks.rados.rados.0.smithi165.stdout:119: done (4 left) 2024-09-16T04:41:55.528 INFO:tasks.rados.rados.0.smithi165.stdout:120: read oid 19 snap 1 2024-09-16T04:41:55.528 INFO:tasks.rados.rados.0.smithi165.stdout:120: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-16T04:41:55.528 INFO:tasks.rados.rados.0.smithi165.stdout:121: rollback oid 17 current snap is 3 2024-09-16T04:41:55.529 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 1 2024-09-16T04:41:55.529 INFO:tasks.rados.rados.0.smithi165.stdout:122: delete oid 18 current snap is 3 2024-09-16T04:41:55.538 INFO:tasks.rados.rados.0.smithi165.stdout:117: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T04:41:55.545 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:55.545 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:55.553 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing rollback tid 0 to smithi16530038-42 2024-09-16T04:41:55.553 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing rollback tid 1 to smithi16530038-42 2024-09-16T04:41:55.553 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing rollback tid 2 to smithi16530038-42 2024-09-16T04:41:55.553 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 49 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:41:55.555 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:55.552+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:55.559 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:41:55.629 INFO:tasks.rados.rados.0.smithi165.stdout:121: finishing rollback tid 0 to smithi16530038-17 2024-09-16T04:41:55.657 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:55.652+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:55.726 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:55.726 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:55.735 INFO:tasks.rados.rados.0.smithi165.stdout:121: finishing rollback tid 1 to smithi16530038-17 2024-09-16T04:41:55.737 INFO:tasks.rados.rados.0.smithi165.stdout:121: finishing rollback tid 2 to smithi16530038-17 2024-09-16T04:41:55.737 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 12 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T04:41:55.741 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:41:55.757 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:55.756+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:instructing pg(s) [3.12s0] on osd.13 to force-backfill; 2024-09-16T04:41:55.762 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-16T04:41:55.835 INFO:tasks.rados.rados.0.smithi165.stdout:120: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-16T04:41:55.858 INFO:tasks.rados.rados.0.smithi165.stdout:117: done (6 left) 2024-09-16T04:41:55.858 INFO:tasks.rados.rados.0.smithi165.stdout:118: done (5 left) 2024-09-16T04:41:55.858 INFO:tasks.rados.rados.0.smithi165.stdout:120: done (4 left) 2024-09-16T04:41:55.858 INFO:tasks.rados.rados.0.smithi165.stdout:121: done (3 left) 2024-09-16T04:41:55.859 INFO:tasks.rados.rados.0.smithi165.stdout:122: done (2 left) 2024-09-16T04:41:55.859 INFO:tasks.rados.rados.0.smithi165.stdout:123: setattr oid 32 current snap is 3 2024-09-16T04:41:55.859 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:41:55.856+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:41:55.862 INFO:tasks.rados.rados.0.smithi165.stdout:124: snap_create 2024-09-16T04:41:55.909 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:55.910 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:55.923 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:41:55.959 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:55.956+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:55.970 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 23 (ObjNum 55 snap 1 seq_num 55) dirty exists 2024-09-16T04:41:56.040 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:56.040 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:56.054 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:41:56.060 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:56.056+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:56.128 INFO:tasks.rados.rados.0.smithi165.stdout:123: done (3 left) 2024-09-16T04:41:56.128 INFO:tasks.rados.rados.0.smithi165.stdout:124: done (2 left) 2024-09-16T04:41:56.128 INFO:tasks.rados.rados.0.smithi165.stdout:125: copy_from oid 7 from oid 37 current snap is 4 2024-09-16T04:41:56.129 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 50 current snap is 4 2024-09-16T04:41:56.129 INFO:tasks.rados.rados.0.smithi165.stdout:126: seq_num 65 ranges {835584=720896,1556480=376832} 2024-09-16T04:41:56.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:41:56.153 INFO:tasks.rados.rados.0.smithi165.stdout:126: writing smithi16530038-50 from 835584 to 1556480 tid 1 2024-09-16T04:41:56.159 INFO:tasks.rados.rados.0.smithi165.stdout:126: writing smithi16530038-50 from 1556480 to 1933312 tid 2 2024-09-16T04:41:56.160 INFO:tasks.rados.rados.0.smithi165.stdout:127: snap_remove snap 3 2024-09-16T04:41:56.160 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:56.156+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:56.174 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:56.175 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:56.189 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:41:56.275 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:56.272+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:56.356 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:56.357 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:56.361 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:56.356+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:56.374 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:41:56.463 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:56.460+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:56.538 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:56.538 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:56.552 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:41:56.596 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:56.592+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:56.664 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:56.660+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:56.719 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:56.720 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:56.733 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:41:56.765 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:56.764+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:56.859 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:56.859 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:56.866 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:56.864+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:56.873 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:41:56.966 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:56.964+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:56.995 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:56.995 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:57.009 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:41:57.067 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:57.063+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:57.082 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:41:57.128 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:57.129 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:57.131 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:127: done (4 left) 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 22 current snap is 4 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:126: finishing write tid 1 to smithi16530038-50 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:126: finishing write tid 2 to smithi16530038-50 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:126: finishing write tid 3 to smithi16530038-50 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:126: finishing write tid 4 to smithi16530038-50 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:126: oid 50 updating version 0 to 7 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:126: oid 50 version 7 is already newer than 6 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:126: oid 50 version 7 is already newer than 5 2024-09-16T04:41:57.136 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 7 (ObjNum 65 snap 4 seq_num 65) dirty exists 2024-09-16T04:41:57.137 INFO:tasks.rados.rados.0.smithi165.stdout:126: left oid 50 (ObjNum 65 snap 4 seq_num 65) 2024-09-16T04:41:57.137 INFO:tasks.rados.rados.0.smithi165.stdout:128: seq_num 66 ranges {360448=737280,1097728=376832} 2024-09-16T04:41:57.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:41:57.154 INFO:tasks.rados.rados.0.smithi165.stdout:128: writing smithi16530038-22 from 360448 to 1097728 tid 1 2024-09-16T04:41:57.161 INFO:tasks.rados.rados.0.smithi165.stdout:128: writing smithi16530038-22 from 1097728 to 1474560 tid 2 2024-09-16T04:41:57.161 INFO:tasks.rados.rados.0.smithi165.stdout:126: done (4 left) 2024-09-16T04:41:57.161 INFO:tasks.rados.rados.0.smithi165.stdout:129: rollback oid 3 current snap is 4 2024-09-16T04:41:57.161 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 1 2024-09-16T04:41:57.161 INFO:tasks.rados.rados.0.smithi165.stdout:130: read oid 21 snap -1 2024-09-16T04:41:57.161 INFO:tasks.rados.rados.0.smithi165.stdout:130: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-16T04:41:57.162 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 38 current snap is 4 2024-09-16T04:41:57.162 INFO:tasks.rados.rados.0.smithi165.stdout:131: seq_num 67 ranges {1605632=753664,2359296=376832} 2024-09-16T04:41:57.167 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:57.164+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:57.196 INFO:tasks.rados.rados.0.smithi165.stdout:131: writing smithi16530038-38 from 1605632 to 2359296 tid 1 2024-09-16T04:41:57.200 INFO:tasks.rados.rados.0.smithi165.stdout:131: writing smithi16530038-38 from 2359296 to 2736128 tid 2 2024-09-16T04:41:57.201 INFO:tasks.rados.rados.0.smithi165.stdout:132: snap_create 2024-09-16T04:41:57.252 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:41:57.252 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:41:57.269 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:57.264+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:57.274 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:41:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:41:57.280 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:41:57.281 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:41:57.295 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:41:57.296 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:41:57.296 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:41:57.318 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:41:57.318 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:41:57.335 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:41:57.335 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:41:57.356 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:41:57.357 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:41:57.369 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:41:57.364+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:41:57.376 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:41:57.376 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:41:57.397 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:41:57.397 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:41:57.419 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:57.419 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:57.419 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:57.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:41:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:131: finishing write tid 1 to smithi16530038-38 2024-09-16T04:41:57.451 INFO:tasks.rados.rados.0.smithi165.stdout:129: finishing rollback tid 0 to smithi16530038-3 2024-09-16T04:41:57.474 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:41:57.474 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:41:57.478 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:57.476+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:57.503 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:41:57.504 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:41:57.522 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:41:57.522 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:41:57.543 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:41:57.543 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:41:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:131: finishing write tid 2 to smithi16530038-38 2024-09-16T04:41:57.548 INFO:tasks.rados.rados.0.smithi165.stdout:131: finishing write tid 3 to smithi16530038-38 2024-09-16T04:41:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:129: finishing rollback tid 1 to smithi16530038-3 2024-09-16T04:41:57.562 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:41:57.562 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:41:57.566 INFO:tasks.rados.rados.0.smithi165.stdout:128: finishing write tid 1 to smithi16530038-22 2024-09-16T04:41:57.566 INFO:tasks.rados.rados.0.smithi165.stdout:128: finishing write tid 2 to smithi16530038-22 2024-09-16T04:41:57.566 INFO:tasks.rados.rados.0.smithi165.stdout:128: finishing write tid 3 to smithi16530038-22 2024-09-16T04:41:57.571 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:57.567+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:57.601 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:57.601 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:41:57.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:41:57.601 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:119 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'81, mlcod=28'78)", 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 24.152389477, 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20259546, 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.602 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.603 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:41:57.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:41:57.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444891+0000", 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6360000000000001e-06 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.445043+0000", 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000151581 2024-09-16T04:41:57.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.445078+0000", 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5077000000000003e-05 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.545224+0000", 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10014579899999999 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.645477+0000", 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100253114 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.645521+0000", 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4218999999999998e-05 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.647416+0000", 2024-09-16T04:41:57.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001895477 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.647485+0000", 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8557000000000001e-05 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.608 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:146 1.0 e29/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 29'104, mlcod=28'102)", 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.183574758999999, 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20228226799999999, 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413706+0000", 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7400000000000001e-06 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413718+0000", 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1892000000000001e-05 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413749+0000", 2024-09-16T04:41:57.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0848e-05 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.513883+0000", 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100134561 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.614137+0000", 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100254093 2024-09-16T04:41:57.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.614197+0000", 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9703000000000001e-05 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.615867+0000", 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001670073 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.615987+0000", 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000119358 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.615 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:168 1.0 e29/23 1:30306672:devicehealth::main.db.0000000000000000:head v 29'122, mlcod=29'119)", 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:36.682449+0000", 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.914830261999999, 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20211510199999999, 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682449+0000", 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682449+0000", 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682451+0000", 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.846e-06 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682570+0000", 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011917399999999999 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.682599+0000", 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9328000000000001e-05 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.782753+0000", 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100153699 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.782936+0000", 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000183606 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.782983+0000", 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6536999999999998e-05 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.884482+0000", 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101499362 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:36.884564+0000", 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1550000000000004e-05 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:202 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'148, mlcod=29'145)", 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.251008138, 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30312661499999999, 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:41:57.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346273+0000", 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7630000000000001e-06 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346284+0000", 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1489e-05 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346311+0000", 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6888000000000001e-05 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346325+0000", 2024-09-16T04:41:57.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3669000000000001e-05 2024-09-16T04:41:57.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.445851+0000", 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099526192999999999 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.445885+0000", 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4075000000000002e-05 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649211+0000", 2024-09-16T04:41:57.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20332629399999999 2024-09-16T04:41:57.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649397+0000", 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018624399999999999 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:203 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'149, mlcod=29'145)", 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.250539965000002, 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30271984099999999, 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.627 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346740+0000", 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0699999999999996e-07 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346874+0000", 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000134585 2024-09-16T04:41:57.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346885+0000", 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0283e-05 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.446978+0000", 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009309199999999 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447153+0000", 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000174721 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447187+0000", 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4913999999999998e-05 2024-09-16T04:41:57.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649408+0000", 2024-09-16T04:41:57.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20222030199999999 2024-09-16T04:41:57.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649459+0000", 2024-09-16T04:41:57.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1137000000000002e-05 2024-09-16T04:41:57.633 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.634 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:204 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'150, mlcod=29'145)", 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.150226653000001, 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246620800000001, 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:41:57.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:41:57.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447054+0000", 2024-09-16T04:41:57.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3430000000000001e-06 2024-09-16T04:41:57.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447066+0000", 2024-09-16T04:41:57.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2055999999999999e-05 2024-09-16T04:41:57.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447085+0000", 2024-09-16T04:41:57.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9165e-05 2024-09-16T04:41:57.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447099+0000", 2024-09-16T04:41:57.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3791999999999999e-05 2024-09-16T04:41:57.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448012+0000", 2024-09-16T04:41:57.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000913319 2024-09-16T04:41:57.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448042+0000", 2024-09-16T04:41:57.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0074e-05 2024-09-16T04:41:57.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649463+0000", 2024-09-16T04:41:57.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20142125599999999 2024-09-16T04:41:57.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.647 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.647 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649519+0000", 2024-09-16T04:41:57.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5203e-05 2024-09-16T04:41:57.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.647 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.648 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:205 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'151, mlcod=29'145)", 2024-09-16T04:41:57.648 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:57.648 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.150162191, 2024-09-16T04:41:57.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202457419, 2024-09-16T04:41:57.649 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.649 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.649 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.649 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:57.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:41:57.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:41:57.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447123+0000", 2024-09-16T04:41:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5600000000000001e-06 2024-09-16T04:41:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447133+0000", 2024-09-16T04:41:57.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7249999999999992e-06 2024-09-16T04:41:57.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447140+0000", 2024-09-16T04:41:57.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.204e-06 2024-09-16T04:41:57.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448504+0000", 2024-09-16T04:41:57.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013644340000000001 2024-09-16T04:41:57.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448533+0000", 2024-09-16T04:41:57.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8691e-05 2024-09-16T04:41:57.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649523+0000", 2024-09-16T04:41:57.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20098977900000001 2024-09-16T04:41:57.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649574+0000", 2024-09-16T04:41:57.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1523000000000001e-05 2024-09-16T04:41:57.659 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.659 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.660 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:206 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'152, mlcod=29'145)", 2024-09-16T04:41:57.660 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:57.660 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.15012909, 2024-09-16T04:41:57.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20247812800000001, 2024-09-16T04:41:57.661 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.661 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.661 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.661 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:57.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:57.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:41:57.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.03e-07 2024-09-16T04:41:57.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447206+0000", 2024-09-16T04:41:57.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5952999999999998e-05 2024-09-16T04:41:57.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447217+0000", 2024-09-16T04:41:57.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0533000000000001e-05 2024-09-16T04:41:57.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447228+0000", 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1012000000000001e-05 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448968+0000", 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001740412 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448996+0000", 2024-09-16T04:41:57.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7331999999999999e-05 2024-09-16T04:41:57.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649579+0000", 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20058309599999999 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649628+0000", 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9387e-05 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.669 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:207 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'153, mlcod=29'145)", 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447828+0000", 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.149451070000001, 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20223788500000001, 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447828+0000", 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447828+0000", 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447829+0000", 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.184e-06 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447838+0000", 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447853+0000", 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5189e-05 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447863+0000", 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0229e-05 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.648746+0000", 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20088268300000001 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.648779+0000", 2024-09-16T04:41:57.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3104e-05 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.650016+0000", 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00123682 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.650066+0000", 2024-09-16T04:41:57.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0049999999999997e-05 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:209 1.0 e32/23 1:30306672:devicehealth::main.db.0000000000000000:head v 32'155, mlcod=30'152)", 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:41:57.675 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.844563993000001, 2024-09-16T04:41:57.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20243324500000001, 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752717+0000", 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.66e-06 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752778+0000", 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1816000000000006e-05 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752805+0000", 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6335000000000001e-05 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.852941+0000", 2024-09-16T04:41:57.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100135847 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.953240+0000", 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10029893300000001 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.953275+0000", 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5091e-05 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.955059+0000", 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001784683 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.955148+0000", 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8880000000000003e-05 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.681 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:106 3.es0 3:741a05b8:::smithi16530038-36:head [append 0~589824,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.361454+0000", 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.235824717, 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30540123299999999, 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:57.682 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 106 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.361454+0000", 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.361454+0000", 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.361459+0000", 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.313e-06 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362586+0000", 2024-09-16T04:41:57.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001127617 2024-09-16T04:41:57.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362619+0000", 2024-09-16T04:41:57.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2814999999999997e-05 2024-09-16T04:41:57.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362635+0000", 2024-09-16T04:41:57.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5534999999999999e-05 2024-09-16T04:41:57.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362791+0000", 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000156391 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.363200+0000", 2024-09-16T04:41:57.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040885900000000002 2024-09-16T04:41:57.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:57.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.365567+0000", 2024-09-16T04:41:57.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0023674949999999998 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.462956+0000", 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097388308000000007 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.666740+0000", 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20378439700000001 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.666856+0000", 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000115503 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:107 3.es0 3:741a05b8:::smithi16530038-36:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:57.691 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362668+0000", 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.234610891999999, 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30452001299999998, 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 107 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362668+0000", 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362668+0000", 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362670+0000", 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.643e-06 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362680+0000", 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8460000000000003e-06 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362699+0000", 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9568999999999998e-05 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362709+0000", 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.367375+0000", 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0046667230000000002 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.367636+0000", 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000261155 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:57.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.368933+0000", 2024-09-16T04:41:57.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012968910000000001 2024-09-16T04:41:57.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:57.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.462959+0000", 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094025238999999997 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.667053+0000", 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20409445500000001 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.667188+0000", 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013508300000000001 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:108 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:41:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.234511288, 2024-09-16T04:41:57.701 INFO:tasks.rados.rados.0.smithi165.stdout:129: finishing rollback tid 2 to smithi16530038-3 2024-09-16T04:41:57.701 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 25 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T04:41:57.701 INFO:tasks.rados.rados.0.smithi165.stdout:128: finishing write tid 4 to smithi16530038-22 2024-09-16T04:41:57.702 INFO:tasks.rados.rados.0.smithi165.stdout:128: oid 22 updating version 0 to 22 2024-09-16T04:41:57.702 INFO:tasks.rados.rados.0.smithi165.stdout:128: oid 22 updating version 22 to 23 2024-09-16T04:41:57.702 INFO:tasks.rados.rados.0.smithi165.stdout:128: oid 22 updating version 23 to 24 2024-09-16T04:41:57.702 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 24 (ObjNum 66 snap 4 seq_num 66) dirty exists 2024-09-16T04:41:57.702 INFO:tasks.rados.rados.0.smithi165.stdout:128: left oid 22 (ObjNum 66 snap 4 seq_num 66) 2024-09-16T04:41:57.702 INFO:tasks.rados.rados.0.smithi165.stdout:130: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-16T04:41:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60644887800000002, 2024-09-16T04:41:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 108 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362769+0000", 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.364e-06 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362776+0000", 2024-09-16T04:41:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9789999999999996e-06 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362795+0000", 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8569000000000001e-05 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362811+0000", 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6084999999999999e-05 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369499+0000", 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066885180000000001 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369745+0000", 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024616900000000002 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669288+0000", 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29954281100000002 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669425+0000", 2024-09-16T04:41:57.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013708200000000001 2024-09-16T04:41:57.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.969217+0000", 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29979130100000001 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 36/32 ECSubWriteReply(tid=13, last_complete=36'8, committed=1, applied=1))", 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.668440+0000", 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.928839203000001, 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.299416507, 2024-09-16T04:41:57.709 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668440+0000", 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668440+0000", 2024-09-16T04:41:57.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668441+0000", 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5999999999999999e-06 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668450+0000", 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2090000000000005e-06 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668479+0000", 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9354999999999999e-05 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.768613+0000", 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100134073 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967795+0000", 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19918244600000001 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967856+0000", 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0823999999999999e-05 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.714 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpReadReply(3.es0 36/32 ECSubReadReply(tid=14, attrs_read=0))", 2024-09-16T04:41:57.714 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.670590+0000", 2024-09-16T04:41:57.714 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.926689056000001, 2024-09-16T04:41:57.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29786822099999999, 2024-09-16T04:41:57.715 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.715 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.715 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:57.715 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670590+0000", 2024-09-16T04:41:57.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670590+0000", 2024-09-16T04:41:57.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670592+0000", 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9530000000000002e-06 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670618+0000", 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6375000000000001e-05 2024-09-16T04:41:57.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670647+0000", 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8296000000000001e-05 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670665+0000", 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8308000000000002e-05 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.968355+0000", 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29769044500000003 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.968458+0000", 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010284400000000001 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 36/32 ECSubWriteReply(tid=13, last_complete=36'8, committed=1, applied=1))", 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.755072+0000", 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.84220732, 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21260926999999999, 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755072+0000", 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755072+0000", 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755073+0000", 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5e-06 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755088+0000", 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5291999999999999e-05 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755111+0000", 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2582000000000002e-05 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755130+0000", 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9165e-05 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967621+0000", 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21249069800000001 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967681+0000", 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0032999999999999e-05 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.0s1 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.975243+0000", 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.622036420000001, 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202203736, 2024-09-16T04:41:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.975243+0000", 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.975243+0000", 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.975245+0000", 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3159999999999999e-06 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.975412+0000", 2024-09-16T04:41:57.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016693099999999999 2024-09-16T04:41:57.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.975450+0000", 2024-09-16T04:41:57.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8371000000000002e-05 2024-09-16T04:41:57.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.733 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.075547+0000", 2024-09-16T04:41:57.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100096719 2024-09-16T04:41:57.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.075683+0000", 2024-09-16T04:41:57.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013643600000000001 2024-09-16T04:41:57.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:57.735 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.075698+0000", 2024-09-16T04:41:57.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4899000000000001e-05 2024-09-16T04:41:57.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.177367+0000", 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101668538 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.177446+0000", 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9525999999999997e-05 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.737 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(osd.3.22:2 3.es0 3:7ee6b8f9:::smithi16530038-8:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:57.737 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.182014+0000", 2024-09-16T04:41:57.737 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 9.4152648600000006, 2024-09-16T04:41:57.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.302278981, 2024-09-16T04:41:57.737 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.738 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.738 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:41:57.738 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182014+0000", 2024-09-16T04:41:57.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182014+0000", 2024-09-16T04:41:57.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182017+0000", 2024-09-16T04:41:57.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2430000000000001e-06 2024-09-16T04:41:57.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182028+0000", 2024-09-16T04:41:57.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0322999999999999e-05 2024-09-16T04:41:57.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.741 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182047+0000", 2024-09-16T04:41:57.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9779000000000002e-05 2024-09-16T04:41:57.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.742 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.742 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182062+0000", 2024-09-16T04:41:57.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4965e-05 2024-09-16T04:41:57.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182177+0000", 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000114394 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182346+0000", 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000169085 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.484293+0000", 2024-09-16T04:41:57.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30194719199999998 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpReadReply(3.es0 38/32 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:41:57.744 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 9.4129148259999997, 2024-09-16T04:41:57.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29996245500000002, 2024-09-16T04:41:57.745 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.745 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.745 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:57.745 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:41:57.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184400+0000", 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5800999999999998e-05 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184413+0000", 2024-09-16T04:41:57.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3169e-05 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.483752+0000", 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29933919199999998 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.484327+0000", 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00057429300000000003 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:281 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:53.099542+0000", 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.4977374010000002, 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30637049, 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:57.750 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 281 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099542+0000", 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099542+0000", 2024-09-16T04:41:57.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099543+0000", 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.569999999999999e-07 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099550+0000", 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4189999999999996e-06 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099565+0000", 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4683e-05 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099577+0000", 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2113000000000001e-05 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.203298+0000", 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103721174 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.203433+0000", 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000134764 2024-09-16T04:41:57.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.405912+0000", 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20247937999999999 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.755 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:57.755 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:41:57.756 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:57.700+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:57.757 INFO:tasks.rados.rados.0.smithi165.stdout:131: finishing write tid 4 to smithi16530038-38 2024-09-16T04:41:57.757 INFO:tasks.rados.rados.0.smithi165.stdout:131: oid 38 updating version 0 to 13 2024-09-16T04:41:57.757 INFO:tasks.rados.rados.0.smithi165.stdout:131: oid 38 updating version 13 to 14 2024-09-16T04:41:57.757 INFO:tasks.rados.rados.0.smithi165.stdout:131: oid 38 updating version 14 to 15 2024-09-16T04:41:57.757 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 15 (ObjNum 67 snap 4 seq_num 67) dirty exists 2024-09-16T04:41:57.757 INFO:tasks.rados.rados.0.smithi165.stdout:131: left oid 38 (ObjNum 67 snap 4 seq_num 67) 2024-09-16T04:41:57.801 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:41:57.800+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:41:57.802 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:41:57.802 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:41:57.827 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:41:57.827 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:41:57.862 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:41:57.862 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:41:57.886 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "replica_scrub(pg: 3.5s6,from:0'0,to:47'15,epoch:47/37,start:3:a0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:57.856515+0000", 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.029632947999999999, 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.029655270000000001, 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856515+0000", 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856515+0000", 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856517+0000", 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.984e-06 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.888 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856529+0000", 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2242000000000001e-05 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856562+0000", 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2821999999999998e-05 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856580+0000", 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7773000000000001e-05 2024-09-16T04:41:57.889 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.890 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:57.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:57.890 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:41:57.890 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 1 2024-09-16T04:41:57.890 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:57.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:41:57.903 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:57.900+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:58.003 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:57.999+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:58.069 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:58.069 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:58.069 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:58.069 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:58.069 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:58.083 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:41:58.104 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:58.100+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:128: done (7 left) 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:129: done (6 left) 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:130: done (5 left) 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:131: done (4 left) 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:132: done (3 left) 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:133: read oid 36 snap 4 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:133: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:134: read oid 1 snap 1 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:134: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:41:58.157 INFO:tasks.rados.rados.0.smithi165.stdout:135: delete oid 23 current snap is 5 2024-09-16T04:41:58.204 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:58.200+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:58.206 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:58.206 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:41:58.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 36/32 ECSubWrite(tid=6, reqid=client.4728.0:118, at_version=36'5, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.756468+0000", 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.448457366, 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102545397, 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756468+0000", 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756468+0000", 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756469+0000", 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6300000000000004e-07 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756729+0000", 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025997800000000001 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756748+0000", 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9018999999999999e-05 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.856849+0000", 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010113399999999 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.857149+0000", 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00030028299999999998 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.857163+0000", 2024-09-16T04:41:58.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3752e-05 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.858932+0000", 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017691759999999999 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.859013+0000", 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1291999999999993e-05 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 36/32 ECSubWrite(tid=7, reqid=client.4728.0:119, at_version=36'6, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.856899+0000", 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.348025862, 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10128643, 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.856899+0000", 2024-09-16T04:41:58.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.856899+0000", 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.856901+0000", 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.277e-06 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.856975+0000", 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4256000000000002e-05 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.856999+0000", 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4488e-05 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.857020+0000", 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0959e-05 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.957105+0000", 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10008485 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.957124+0000", 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8402000000000001e-05 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.958108+0000", 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00098438299999999992 2024-09-16T04:41:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.958186+0000", 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7815e-05 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/32 ECSubWrite(tid=161, reqid=client.4728.0:127, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.862409+0000", 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.342516291999999, 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10371385299999999, 2024-09-16T04:41:58.216 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.862409+0000", 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.862409+0000", 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.862411+0000", 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4600000000000002e-06 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.863037+0000", 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00062557300000000004 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.863071+0000", 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4214999999999997e-05 2024-09-16T04:41:58.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.863088+0000", 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7299000000000002e-05 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.963347+0000", 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100258786 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.963372+0000", 2024-09-16T04:41:58.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5168999999999999e-05 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.966059+0000", 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026868450000000002 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.966123+0000", 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3505999999999998e-05 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/32 ECSubWrite(tid=162, reqid=client.4728.0:128, at_version=36'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.865257+0000", 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.339668432, 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101112959, 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.865257+0000", 2024-09-16T04:41:58.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.865257+0000", 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.865258+0000", 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.403e-06 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.865286+0000", 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7787999999999999e-05 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.865311+0000", 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4922999999999998e-05 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.865325+0000", 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4065e-05 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.964699+0000", 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099374167999999999 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.964714+0000", 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4616e-05 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.966331+0000", 2024-09-16T04:41:58.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016168580000000001 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.966370+0000", 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9137999999999997e-05 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.as5 36/32 ECSubWrite(tid=16, reqid=client.4728.0:171, at_version=36'9, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.898522+0000", 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.306403403000001, 2024-09-16T04:41:58.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10165025799999999, 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898522+0000", 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898522+0000", 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898524+0000", 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.57e-06 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898566+0000", 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1284e-05 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898600+0000", 2024-09-16T04:41:58.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3940999999999999e-05 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898619+0000", 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9289000000000001e-05 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.998828+0000", 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100208912 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.998851+0000", 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2838000000000001e-05 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.000097+0000", 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012462020000000001 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.000172+0000", 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5222000000000006e-05 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.229 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/32 ECSubWrite(tid=35, reqid=client.4728.0:169, at_version=0'0, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:47.777214+0000", 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.427711309999999, 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10154481, 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.777214+0000", 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.777214+0000", 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.777216+0000", 2024-09-16T04:41:58.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8160000000000001e-06 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.777237+0000", 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1369000000000002e-05 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.777269+0000", 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1435e-05 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.777284+0000", 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5486e-05 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.877539+0000", 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100255222 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.877553+0000", 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4042999999999999e-05 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.878676+0000", 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011228639999999999 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.878759+0000", 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2575000000000001e-05 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 38/32 ECSubWrite(tid=34, reqid=client.4728.0:182, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:58.234 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.276943+0000", 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 9.9279817480000006, 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10333068199999999, 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.276943+0000", 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.276943+0000", 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.276945+0000", 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3799999999999999e-06 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.277066+0000", 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0001209 2024-09-16T04:41:58.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.277104+0000", 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8708000000000003e-05 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.277129+0000", 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4085999999999999e-05 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.377334+0000", 2024-09-16T04:41:58.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100205586 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.377349+0000", 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4666e-05 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.380172+0000", 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0028237119999999999 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.380274+0000", 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000101644 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 38/32 ECSubWrite(tid=35, reqid=client.4728.0:183, at_version=38'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.279231+0000", 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 9.9256943639999999, 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102482004, 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.239 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279231+0000", 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279231+0000", 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279232+0000", 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.593e-06 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279264+0000", 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1535000000000002e-05 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279302+0000", 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7914999999999999e-05 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279326+0000", 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4215000000000001e-05 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.379685+0000", 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100358959 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.379710+0000", 2024-09-16T04:41:58.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4708000000000001e-05 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.381607+0000", 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001896752 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.381713+0000", 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000106327 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 38/32 ECSubWrite(tid=42, reqid=client.4728.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.705545+0000", 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 9.4993806630000002, 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101363042, 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.705545+0000", 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.705545+0000", 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.705546+0000", 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.28e-06 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.705563+0000", 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7504999999999999e-05 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.705587+0000", 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3468999999999998e-05 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.805732+0000", 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100145495 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.805888+0000", 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015597199999999999 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.805903+0000", 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4484999999999999e-05 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.806827+0000", 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00092405700000000005 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.806908+0000", 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0778999999999993e-05 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.248 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.7950708459999998, 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20191694600000001, 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409856+0000", 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409917+0000", 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1023999999999997e-05 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409958+0000", 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0945000000000002e-05 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510088+0000", 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100130017 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510247+0000", 2024-09-16T04:41:58.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015866599999999999 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510287+0000", 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0330999999999997e-05 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611689+0000", 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101401328 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611771+0000", 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2582000000000003e-05 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=31, reqid=client.4728.0:186, at_version=42'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.510322+0000", 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.6946029009999997, 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101514491, 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.253 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510322+0000", 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510322+0000", 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510323+0000", 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.08e-06 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510368+0000", 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4203999999999999e-05 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510396+0000", 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8569e-05 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510419+0000", 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2728999999999999e-05 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.511663+0000", 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001243919 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.511679+0000", 2024-09-16T04:41:58.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5763000000000001e-05 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611781+0000", 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100102683 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611837+0000", 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5544000000000002e-05 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=33, reqid=client.4728.0:187, at_version=42'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.510448+0000", 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.694476828, 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10143498200000001, 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510448+0000", 2024-09-16T04:41:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510448+0000", 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510449+0000", 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510526+0000", 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7063000000000004e-05 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510557+0000", 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0247999999999999e-05 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510576+0000", 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9799000000000001e-05 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.512155+0000", 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015791100000000001 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.512168+0000", 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.204e-05 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611841+0000", 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099673903999999994 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611883+0000", 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1931000000000002e-05 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPush(3.17s1 42/41 [PushOp(3:ed82f61a:::smithi16530038-1:head, version: 40'4, data_included: [0~151552], data_size: 151552, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed82f61a:::smithi16530038-1:head@40'4, size: 606208, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 606208, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.831178+0000", 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.373746863, 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102367006, 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.262 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831178+0000", 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831178+0000", 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831180+0000", 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.513e-06 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831696+0000", 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00051605900000000003 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831742+0000", 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5951000000000001e-05 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.931903+0000", 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.1001611 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.932153+0000", 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025006099999999999 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.933545+0000", 2024-09-16T04:41:58.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013923220000000001 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=39, reqid=client.4728.0:187, at_version=0'0, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.923239+0000", 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.2816863749999996, 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101838922, 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923239+0000", 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923239+0000", 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923241+0000", 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8139999999999999e-06 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923264+0000", 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3142e-05 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923340+0000", 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6095999999999998e-05 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923398+0000", 2024-09-16T04:41:58.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7883999999999997e-05 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.023680+0000", 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100282702 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.023695+0000", 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4656e-05 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.024995+0000", 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001299929 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.025078+0000", 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2699000000000002e-05 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s6 44/37 ECSubWrite(tid=11, reqid=osd.5.0:11, at_version=44'8, trim_to=0'0, roll_forward_to=42'6))", 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:41:58.270 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.0868260899999997, 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10484916599999999, 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118100+0000", 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9599999999999998e-07 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118155+0000", 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4663999999999997e-05 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118199+0000", 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4861000000000003e-05 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118225+0000", 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5415999999999999e-05 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118379+0000", 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015390299999999999 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118406+0000", 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7317999999999998e-05 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222865+0000", 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10445916099999999 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222948+0000", 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3047000000000006e-05 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 44/41 ECSubWrite(tid=40, reqid=osd.14.0:40, at_version=44'10, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.0803245510000004, 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10233370999999999, 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:41:58.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8999999999999996e-07 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.276 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124656+0000", 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4641000000000001e-05 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124686+0000", 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9872000000000001e-05 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224794+0000", 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100107847 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224954+0000", 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000160163 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224980+0000", 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6162000000000001e-05 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.226840+0000", 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001859793 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.226934+0000", 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4542000000000004e-05 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.12s3 44/41 ECSubWrite(tid=179, reqid=osd.13.0:179, at_version=44'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 3.9837113230000001, 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103206719, 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.279 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4000000000000002e-07 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221253+0000", 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9118000000000001e-05 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221277+0000", 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3828999999999998e-05 2024-09-16T04:41:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221293+0000", 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5502000000000002e-05 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.321532+0000", 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100239168 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.321562+0000", 2024-09-16T04:41:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9884999999999999e-05 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.324333+0000", 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027712380000000001 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.324421+0000", 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7539000000000003e-05 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDScrubReserve(3.12s3,QREQUEST,e:47,from:13(0),reservation_nonce:5)", 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:57.435382+0000", 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.76954363000000003, 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10033275899999999, 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.435382+0000", 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.435382+0000", 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.435384+0000", 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.734e-06 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.435397+0000", 2024-09-16T04:41:58.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2302e-05 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.435428+0000", 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1458999999999998e-05 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.535541+0000", 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10011334400000001 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.286 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.535628+0000", 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7082000000000004e-05 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.535658+0000", 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.932e-05 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.535714+0000", 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6518000000000003e-05 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.287 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 47/37 ECSubWrite(tid=23, reqid=client.4728.0:346, at_version=0'0, trim_to=0'0, roll_forward_to=47'15))", 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:57.548741+0000", 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.656184455, 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10116015, 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.548741+0000", 2024-09-16T04:41:58.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.548741+0000", 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.548743+0000", 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.756e-06 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.548760+0000", 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7149000000000001e-05 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.548781+0000", 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1297999999999998e-05 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.548793+0000", 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2228999999999999e-05 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.649002+0000", 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100208821 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.649011+0000", 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6130000000000004e-06 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.649836+0000", 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000825035 2024-09-16T04:41:58.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.649901+0000", 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5248999999999998e-05 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "replica_scrub(pg: 3.5s6,from:0'0,to:47'15,epoch:47/37,start:3:a0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:57.856515+0000", 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.348410149, 2024-09-16T04:41:58.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10029874799999999, 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856515+0000", 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856515+0000", 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856517+0000", 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.984e-06 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856529+0000", 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2242000000000001e-05 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856562+0000", 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2821999999999998e-05 2024-09-16T04:41:58.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.856580+0000", 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7773000000000001e-05 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.956762+0000", 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10018188 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:58.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.956773+0000", 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0985000000000001e-05 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:57.956814+0000", 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1062000000000001e-05 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.296 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:58.297 INFO:tasks.rados.rados.0.smithi165.stdout:135: done (5 left) 2024-09-16T04:41:58.298 INFO:tasks.rados.rados.0.smithi165.stdout:136: rollback oid 20 current snap is 5 2024-09-16T04:41:58.298 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 1 2024-09-16T04:41:58.298 INFO:tasks.rados.rados.0.smithi165.stdout:137: read oid 14 snap -1 2024-09-16T04:41:58.298 INFO:tasks.rados.rados.0.smithi165.stdout:137: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:41:58.298 INFO:tasks.rados.rados.0.smithi165.stdout:138: setattr oid 15 current snap is 5 2024-09-16T04:41:58.298 INFO:tasks.rados.rados.0.smithi165.stdout:133: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T04:41:58.299 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:41:58.304 INFO:tasks.rados.rados.0.smithi165.stdout:133: done (7 left) 2024-09-16T04:41:58.304 INFO:tasks.rados.rados.0.smithi165.stdout:139: delete oid 11 current snap is 5 2024-09-16T04:41:58.305 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:58.300+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:58.406 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:58.403+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:41:58.471 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:58.471 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:41:58.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 48/47 ECSubWrite(tid=50, reqid=client.4728.0:308, at_version=48'23, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.112516713, 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11253425, 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358592+0000", 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1540000000000001e-06 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358618+0000", 2024-09-16T04:41:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5622e-05 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358639+0000", 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1288e-05 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358654+0000", 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4278e-05 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360496+0000", 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018424470000000001 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360504+0000", 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4100000000000008e-06 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 48/47 ECSubWrite(tid=51, reqid=client.4728.0:309, at_version=48'24, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:41:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.110726985, 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.110782306, 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360382+0000", 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.004e-06 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360399+0000", 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7278e-05 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360415+0000", 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6004000000000001e-05 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360423+0000", 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1820000000000006e-06 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:41:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 2 2024-09-16T04:41:58.479 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:58.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:41:58.507 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:58.504+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:58.560 INFO:tasks.rados.rados.0.smithi165.stdout:134: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:41:58.596 INFO:tasks.rados.rados.0.smithi165.stdout:114: finishing write tid 4 to smithi16530038-47 2024-09-16T04:41:58.596 INFO:tasks.rados.rados.0.smithi165.stdout:114: oid 47 updating version 0 to 20 2024-09-16T04:41:58.596 INFO:tasks.rados.rados.0.smithi165.stdout:114: oid 47 version 20 is already newer than 19 2024-09-16T04:41:58.596 INFO:tasks.rados.rados.0.smithi165.stdout:114: oid 47 version 20 is already newer than 18 2024-09-16T04:41:58.596 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 20 (ObjNum 61 snap 3 seq_num 61) dirty exists 2024-09-16T04:41:58.596 INFO:tasks.rados.rados.0.smithi165.stdout:114: left oid 47 (ObjNum 61 snap 3 seq_num 61) 2024-09-16T04:41:58.597 INFO:tasks.rados.rados.0.smithi165.stdout:115: finishing rollback tid 1 to smithi16530038-27 2024-09-16T04:41:58.597 INFO:tasks.rados.rados.0.smithi165.stdout:137: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:41:58.604 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:58.604 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:58.604 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:58.604 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:58.604 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:58.607 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:41:58.599+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:41:58.618 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:41:58.660 INFO:tasks.rados.rados.0.smithi165.stdout:114: done (7 left) 2024-09-16T04:41:58.660 INFO:tasks.rados.rados.0.smithi165.stdout:134: done (6 left) 2024-09-16T04:41:58.660 INFO:tasks.rados.rados.0.smithi165.stdout:137: done (5 left) 2024-09-16T04:41:58.660 INFO:tasks.rados.rados.0.smithi165.stdout:139: done (4 left) 2024-09-16T04:41:58.660 INFO:tasks.rados.rados.0.smithi165.stdout:140: snap_remove snap 4 2024-09-16T04:41:58.708 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:41:58.704+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:41:58.741 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpRead(3.cs3 36/32 ECSubRead(tid=24, to_read={3:372a9655:::smithi16530038-32:head=0,4096,0}, subchunks={3:372a9655:::smithi16530038-32:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.967149+0000", 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.770730761999999, 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.200811252, 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.742 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967149+0000", 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967149+0000", 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967152+0000", 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0940000000000002e-06 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967169+0000", 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7549000000000001e-05 2024-09-16T04:41:58.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967212+0000", 2024-09-16T04:41:58.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2701000000000003e-05 2024-09-16T04:41:58.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967243+0000", 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1015e-05 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.167603+0000", 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20036040099999999 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.167961+0000", 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000357492 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=17, reqid=client.4728.0:139, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.651029388, 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.203994169, 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.747 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:41:58.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086853+0000", 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6000000000000001e-06 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087506+0000", 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00065279900000000002 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087552+0000", 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5840999999999997e-05 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087581+0000", 2024-09-16T04:41:58.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9354999999999999e-05 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187852+0000", 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100270242 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187866+0000", 2024-09-16T04:41:58.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4286e-05 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290791+0000", 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102925166 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290845+0000", 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3879999999999999e-05 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=18, reqid=client.4728.0:140, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.648343983, 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27450581499999999, 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089538+0000", 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0370000000000001e-06 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089605+0000", 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6268999999999999e-05 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089663+0000", 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8257000000000001e-05 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.189764+0000", 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100101525 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290404+0000", 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10063934500000001 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290428+0000", 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4782000000000001e-05 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363982+0000", 2024-09-16T04:41:58.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073553303 2024-09-16T04:41:58.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364042+0000", 2024-09-16T04:41:58.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0297000000000002e-05 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 36/32 ECSubWrite(tid=27, reqid=client.4728.0:136, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.162782+0000", 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.575097754, 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20118729299999999, 2024-09-16T04:41:58.759 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.162782+0000", 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.162782+0000", 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.162785+0000", 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163121+0000", 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00033597100000000001 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163151+0000", 2024-09-16T04:41:58.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0787999999999997e-05 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163167+0000", 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6008999999999999e-05 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.188993+0000", 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.025825142999999998 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.189015+0000", 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2498999999999999e-05 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363899+0000", 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.17488398499999999 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363970+0000", 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0702000000000002e-05 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.763 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 36/32 ECSubWrite(tid=29, reqid=client.4728.0:137, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.163851+0000", 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.574029646, 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20078317000000001, 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163851+0000", 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163851+0000", 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163852+0000", 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7430000000000001e-06 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163889+0000", 2024-09-16T04:41:58.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6275000000000001e-05 2024-09-16T04:41:58.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163924+0000", 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5131e-05 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163946+0000", 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1829000000000001e-05 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363543+0000", 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19959737499999999 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363557+0000", 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4348999999999999e-05 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364556+0000", 2024-09-16T04:41:58.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099826600000000004 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364634+0000", 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8201999999999994e-05 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/32 ECSubWrite(tid=32, reqid=client.4728.0:209, at_version=38'17, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.262819159999999, 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202405108, 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475063+0000", 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.906e-06 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475102+0000", 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9195e-05 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475132+0000", 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9861000000000001e-05 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475148+0000", 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.611e-05 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.575413+0000", 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026479000000001 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.575431+0000", 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7912000000000001e-05 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677395+0000", 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101964155 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677466+0000", 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1179000000000004e-05 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.774 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/32 ECSubWrite(tid=33, reqid=client.4728.0:210, at_version=38'18, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.475188+0000", 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.262691907000001, 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202351791, 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475188+0000", 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475188+0000", 2024-09-16T04:41:58.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475189+0000", 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4600000000000004e-07 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475217+0000", 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8388999999999999e-05 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475231+0000", 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3336000000000001e-05 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475243+0000", 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2245999999999999e-05 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.576236+0000", 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10099335800000001 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.576265+0000", 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8279000000000001e-05 2024-09-16T04:41:58.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.779 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677483+0000", 2024-09-16T04:41:58.779 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101218777 2024-09-16T04:41:58.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677540+0000", 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6759999999999999e-05 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/32 ECSubWrite(tid=34, reqid=client.4728.0:211, at_version=38'19, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.486942+0000", 2024-09-16T04:41:58.780 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.250937775000001, 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19185692900000001, 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486942+0000", 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486942+0000", 2024-09-16T04:41:58.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486944+0000", 2024-09-16T04:41:58.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.697e-06 2024-09-16T04:41:58.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486979+0000", 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4623999999999999e-05 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.487008+0000", 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9166000000000002e-05 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.487028+0000", 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9749e-05 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.676989+0000", 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18996172 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677016+0000", 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6804999999999998e-05 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.678706+0000", 2024-09-16T04:41:58.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016896960000000001 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.678799+0000", 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3472000000000002e-05 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.19s6 42/41 ECSubWrite(tid=35, reqid=client.4728.0:262, at_version=42'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.516159+0000", 2024-09-16T04:41:58.787 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.2217209540000002, 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102865605, 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516159+0000", 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516159+0000", 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516161+0000", 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9630000000000001e-06 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516215+0000", 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4041e-05 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516251+0000", 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5747999999999997e-05 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516275+0000", 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4442000000000001e-05 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516645+0000", 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00036991200000000002 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.516669+0000", 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3849000000000002e-05 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.618935+0000", 2024-09-16T04:41:58.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102266151 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.619025+0000", 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9499000000000005e-05 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 44/32 ECSubWrite(tid=56, reqid=osd.11.0:56, at_version=44'19, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.118905+0000", 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.6189747839999997, 2024-09-16T04:41:58.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103954818, 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118905+0000", 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118905+0000", 2024-09-16T04:41:58.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118906+0000", 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3300000000000002e-07 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118962+0000", 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5994e-05 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.794 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118993+0000", 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0442000000000001e-05 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.119017+0000", 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4946000000000001e-05 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.119225+0000", 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020762299999999999 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.119252+0000", 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6846e-05 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222748+0000", 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103495693 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222860+0000", 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011264099999999999 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.11s4 44/41 ECSubWrite(tid=55, reqid=osd.4.0:55, at_version=44'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.213827+0000", 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.5240531900000001, 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106368214, 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.797 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213827+0000", 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213827+0000", 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.798 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213828+0000", 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4000000000000002e-07 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213892+0000", 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4057e-05 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213919+0000", 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7512e-05 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213944+0000", 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4399000000000001e-05 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224385+0000", 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010440974 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224408+0000", 2024-09-16T04:41:58.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3603000000000001e-05 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.320130+0000", 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.095721732000000004 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.320195+0000", 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5396999999999997e-05 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/32 ECSubWrite(tid=62, reqid=osd.7.0:61, at_version=0'0, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.2308221819999998, 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20295831, 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:41:58.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507061+0000", 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5059999999999998e-06 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507088+0000", 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7764999999999999e-05 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507122+0000", 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3671999999999999e-05 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607262+0000", 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100139588 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707564+0000", 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100302577 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707595+0000", 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0482000000000001e-05 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.709942+0000", 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002347671 2024-09-16T04:41:58.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.710016+0000", 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4048999999999995e-05 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/32 ECSubWrite(tid=64, reqid=osd.7.0:64, at_version=44'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.607437+0000", 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.1304428089999998, 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103718054, 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.806 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607437+0000", 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607437+0000", 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607438+0000", 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607476+0000", 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8052000000000001e-05 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607489+0000", 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3216e-05 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707624+0000", 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100134368 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.709037+0000", 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014127740000000001 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.709061+0000", 2024-09-16T04:41:58.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4505999999999999e-05 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.711089+0000", 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020283549999999999 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.711155+0000", 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6026999999999997e-05 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.19s6 44/41 ECSubWrite(tid=42, reqid=client.4728.0:303, at_version=44'18, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.625445+0000", 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.1124349840000001, 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103863684, 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625445+0000", 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625445+0000", 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625448+0000", 2024-09-16T04:41:58.812 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4099999999999998e-06 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625828+0000", 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00038062300000000002 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625872+0000", 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4174000000000001e-05 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625897+0000", 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4895000000000001e-05 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726583+0000", 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100685493 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726602+0000", 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9208999999999999e-05 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.729228+0000", 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026255979999999998 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.729309+0000", 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1281999999999999e-05 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.19s6 44/41 ECSubWrite(tid=43, reqid=client.4728.0:304, at_version=44'19, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.625959+0000", 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.1119217040000002, 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103927117, 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625959+0000", 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625959+0000", 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.625960+0000", 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3400000000000001e-06 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626124+0000", 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000164162 2024-09-16T04:41:58.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626161+0000", 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6995000000000001e-05 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626181+0000", 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9908999999999999e-05 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.728016+0000", 2024-09-16T04:41:58.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101834878 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.728030+0000", 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4577e-05 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.729832+0000", 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018019030000000001 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.729886+0000", 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3353000000000002e-05 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.19s6 44/41 ECSubWrite(tid=44, reqid=client.4728.0:305, at_version=44'20, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.626223+0000", 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.1116573900000004, 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.104551745, 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.820 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626223+0000", 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626223+0000", 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626224+0000", 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.043e-06 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626257+0000", 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3161999999999999e-05 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626278+0000", 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1219000000000001e-05 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.626295+0000", 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7195999999999999e-05 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.728835+0000", 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102539169 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.728858+0000", 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3382000000000001e-05 2024-09-16T04:41:58.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.730703+0000", 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001845441 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.730775+0000", 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1132999999999998e-05 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.825 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 48/47 ECSubWrite(tid=50, reqid=client.4728.0:308, at_version=48'23, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.825 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:41:58.825 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.37928906099999998, 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19739056699999999, 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358592+0000", 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1540000000000001e-06 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358618+0000", 2024-09-16T04:41:58.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5622e-05 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358639+0000", 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1288e-05 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.828 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358654+0000", 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4278e-05 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360496+0000", 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018424470000000001 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360504+0000", 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4100000000000008e-06 2024-09-16T04:41:58.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555946+0000", 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.195441898 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555982+0000", 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5469999999999998e-05 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.830 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 48/47 ECSubWrite(tid=51, reqid=client.4728.0:309, at_version=48'24, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.37749933299999999, 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.196218006, 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360382+0000", 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.004e-06 2024-09-16T04:41:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360399+0000", 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7278e-05 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360415+0000", 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6004000000000001e-05 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360423+0000", 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1820000000000006e-06 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555757+0000", 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19533410100000001 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555766+0000", 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5590000000000006e-06 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.556544+0000", 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00077832900000000002 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.556599+0000", 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4549000000000002e-05 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/32 ECSubWrite(tid=76, reqid=client.4728.0:357, at_version=48'22, trim_to=0'0, roll_forward_to=48'19))", 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.563226+0000", 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.17465403099999999, 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103572995, 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563226+0000", 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563226+0000", 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563227+0000", 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7300000000000005e-07 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563287+0000", 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9982999999999998e-05 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563302+0000", 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5259000000000001e-05 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563314+0000", 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2147000000000001e-05 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.663457+0000", 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100142961 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.663465+0000", 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.892e-06 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.666743+0000", 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0032773730000000001 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.666799+0000", 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6507e-05 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/32 ECSubWrite(tid=77, reqid=client.4728.0:358, at_version=48'23, trim_to=0'0, roll_forward_to=48'19))", 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.563345+0000", 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.17453504, 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103729508, 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:58.841 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563345+0000", 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563345+0000", 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563346+0000", 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7000000000000005e-07 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563365+0000", 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8889000000000002e-05 2024-09-16T04:41:58.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563373+0000", 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3410000000000003e-06 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.563378+0000", 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4049999999999999e-06 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:58.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.663736+0000", 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100357714 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.663746+0000", 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5799999999999998e-06 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.667045+0000", 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0032993710000000002 2024-09-16T04:41:58.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.667075+0000", 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9638e-05 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:58.846 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:58.848 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:41:58.849 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:58.844+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:58.949 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:41:58.948+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:41:59.016 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:59.016 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:59.016 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:41:59.016 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:59.030 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:41:59.050 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:41:59.047+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:41:59.151 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:41:59.147+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:41:59.151 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:41:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:41:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:41:59.152 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:140: done (4 left) 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:141: setattr oid 36 current snap is 5 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 17 (ObjNum 129 snap 0 seq_num 1783810664) dirty exists 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:115: finishing rollback tid 2 to smithi16530038-27 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 24 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing rollback tid 0 to smithi16530038-20 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing rollback tid 1 to smithi16530038-20 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing rollback tid 2 to smithi16530038-20 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 23 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T04:41:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:125: finishing copy_from racing read to smithi16530038-7 2024-09-16T04:41:59.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:41:59.158 INFO:tasks.rados.rados.0.smithi165.stdout:115: done (4 left) 2024-09-16T04:41:59.158 INFO:tasks.rados.rados.0.smithi165.stdout:136: done (3 left) 2024-09-16T04:41:59.159 INFO:tasks.rados.rados.0.smithi165.stdout:138: done (2 left) 2024-09-16T04:41:59.159 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 3 current snap is 5 2024-09-16T04:41:59.159 INFO:tasks.rados.rados.0.smithi165.stdout:142: seq_num 70 ranges {49152=802816,851968=376832} 2024-09-16T04:41:59.166 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:41:59.173 INFO:tasks.rados.rados.0.smithi165.stdout:142: writing smithi16530038-3 from 49152 to 851968 tid 1 2024-09-16T04:41:59.180 INFO:tasks.rados.rados.0.smithi165.stdout:142: writing smithi16530038-3 from 851968 to 1228800 tid 2 2024-09-16T04:41:59.180 INFO:tasks.rados.rados.0.smithi165.stdout:143: read oid 39 snap 1 2024-09-16T04:41:59.180 INFO:tasks.rados.rados.0.smithi165.stdout:143: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T04:41:59.180 INFO:tasks.rados.rados.0.smithi165.stdout:144: rollback oid 48 current snap is 5 2024-09-16T04:41:59.180 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 1 2024-09-16T04:41:59.180 INFO:tasks.rados.rados.0.smithi165.stdout:145: delete oid 34 current snap is 5 2024-09-16T04:41:59.252 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:59.247+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:59.273 INFO:tasks.rados.rados.0.smithi165.stdout:125: finishing copy_from to smithi16530038-7 2024-09-16T04:41:59.273 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:156 3.9s0 3:9de467d5:::smithi16530038-27:head [append 0~475136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.383392+0000", 2024-09-16T04:41:59.288 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.902813618, 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40721533599999998, 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 156 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383392+0000", 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383392+0000", 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383395+0000", 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5560000000000002e-06 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383797+0000", 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040218300000000002 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383823+0000", 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6216000000000001e-05 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.483957+0000", 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10013338300000001 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484413+0000", 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00045616999999999999 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484595+0000", 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000182566 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.485946+0000", 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001351196 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589202+0000", 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10325537 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790481+0000", 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20127900600000001 2024-09-16T04:41:59.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790608+0000", 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012668999999999999 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:157 3.9s0 3:9de467d5:::smithi16530038-27:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.802198293, 2024-09-16T04:41:59.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51218579099999995, 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 157 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:41:59.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484010+0000", 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0329999999999998e-06 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484272+0000", 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026256800000000002 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484296+0000", 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3509000000000001e-05 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484308+0000", 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2699e-05 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584144+0000", 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099835667000000003 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584304+0000", 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016037700000000001 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.585781+0000", 2024-09-16T04:41:59.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001476455 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589205+0000", 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0034239499999999998 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996077+0000", 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40687225300000002 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.299 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996193+0000", 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011628 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:158 3.9s0 3:9de467d5:::smithi16530038-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.801874765000001, 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51808159200000004, 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.300 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 158 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484332+0000", 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484339+0000", 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8630000000000002e-06 2024-09-16T04:41:59.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484355+0000", 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6473e-05 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484364+0000", 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8950000000000004e-06 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.586917+0000", 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102553035 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.587177+0000", 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026005499999999998 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.588403+0000", 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001225152 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589911+0000", 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015086260000000001 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002165+0000", 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41225355800000002 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002413+0000", 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024783200000000002 2024-09-16T04:41:59.305 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.9s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.801805956000001, 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.62506688499999996, 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.306 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484401+0000", 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484405+0000", 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.827e-06 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484416+0000", 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1609e-05 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484425+0000", 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790928+0000", 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30650327300000002 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.791067+0000", 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013860200000000001 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003346+0000", 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212279577 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003611+0000", 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026456899999999998 2024-09-16T04:41:59.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109440+0000", 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105828857 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109467+0000", 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6900999999999999e-05 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 36/32 ECSubWriteReply(tid=19, last_complete=36'4, committed=1, applied=1))", 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.499445+0000", 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.786760485, 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28634094100000002, 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499445+0000", 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499445+0000", 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499447+0000", 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2529999999999999e-06 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499453+0000", 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4570000000000002e-06 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499483+0000", 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9850999999999999e-05 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499515+0000", 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2064999999999999e-05 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.785726+0000", 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.286210878 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.785786+0000", 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0436999999999998e-05 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.19s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:41:59.315 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.178853349000001, 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.976424197, 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107354+0000", 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9829999999999999e-06 2024-09-16T04:41:59.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107362+0000", 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6129999999999998e-06 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107388+0000", 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5675000000000001e-05 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207482+0000", 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009412299999999 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207644+0000", 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000161741 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207649+0000", 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0359999999999997e-06 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082558+0000", 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8749096240000001 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082784+0000", 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022614900000000001 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.083777+0000", 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099225300000000002 2024-09-16T04:41:59.320 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:178 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.208091923, 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1069151189999999, 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.321 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 178 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:41:59.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078116+0000", 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078127+0000", 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.062e-05 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078151+0000", 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4304999999999999e-05 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078175+0000", 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4006e-05 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776496+0000", 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69832100500000005 2024-09-16T04:41:59.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776503+0000", 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.533e-06 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185014+0000", 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40851148700000001 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185029+0000", 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4780000000000001e-05 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:179 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.207996561, 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1068670979999999, 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.326 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 179 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078211+0000", 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.44e-06 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078220+0000", 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:41:59.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078244+0000", 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3867999999999999e-05 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078260+0000", 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6203000000000001e-05 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780436+0000", 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70217526500000005 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780441+0000", 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4620000000000004e-06 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185067+0000", 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40462590799999998 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185076+0000", 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3219999999999999e-06 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:180 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.207923505, 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.106828843, 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.331 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 180 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078284+0000", 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.333 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078291+0000", 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.657e-06 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078310+0000", 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078325+0000", 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4772e-05 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981190+0000", 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90286589500000003 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981194+0000", 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0790000000000002e-06 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185100+0000", 2024-09-16T04:41:59.335 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20390586399999999 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185111+0000", 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0794e-05 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:185 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:41:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.187607526000001, 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98955797199999995, 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098600+0000", 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098611+0000", 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0423e-05 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098642+0000", 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1532000000000002e-05 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098658+0000", 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5996000000000001e-05 2024-09-16T04:41:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683726+0000", 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58506727400000003 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683731+0000", 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088142+0000", 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40441113000000001 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088156+0000", 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4148000000000001e-05 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:186 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:41:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.187519471, 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.089963279, 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 186 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098687+0000", 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.063e-06 2024-09-16T04:41:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098693+0000", 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6230000000000002e-06 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098705+0000", 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2019e-05 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198804+0000", 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098513 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683799+0000", 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48499579100000001 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683801+0000", 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0889999999999998e-06 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188641+0000", 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50483997300000005 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188650+0000", 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.208e-06 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:187 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.087370068, 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98988477799999997, 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 187 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198838+0000", 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.86e-06 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198846+0000", 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0059999999999992e-06 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198864+0000", 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8468999999999999e-05 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198876+0000", 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1443e-05 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783957+0000", 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58508126900000001 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783962+0000", 2024-09-16T04:41:59.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7230000000000001e-06 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188710+0000", 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40474714899999997 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188721+0000", 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0859000000000001e-05 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s4 38/37 ECSubWrite(tid=5, reqid=client.4728.0:189, at_version=38'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.701042+0000", 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.585163424999999, 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38743471499999999, 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.352 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701042+0000", 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701042+0000", 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701044+0000", 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.03e-06 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701364+0000", 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032004700000000001 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701407+0000", 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2710000000000003e-05 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701433+0000", 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5409e-05 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.356 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.983563+0000", 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.282130618 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.983572+0000", 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088375+0000", 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.104802627 2024-09-16T04:41:59.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088477+0000", 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010224499999999999 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 42/41 ECSubWriteReply(tid=29, last_complete=42'6, committed=1, applied=1))", 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.510381+0000", 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.775824321, 2024-09-16T04:41:59.358 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206954525, 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510381+0000", 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510381+0000", 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510383+0000", 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9999999999999999e-06 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510394+0000", 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0743e-05 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510434+0000", 2024-09-16T04:41:59.360 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0048e-05 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610580+0000", 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100145578 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.717270+0000", 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106690259 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.361 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.717336+0000", 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5896999999999996e-05 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s4 42/37 ECSubWrite(tid=9, reqid=client.4728.0:269, at_version=42'6, trim_to=0'0, roll_forward_to=38'4))", 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.521672+0000", 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.7645336250000003, 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29629296900000002, 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.362 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.521672+0000", 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.521672+0000", 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.521674+0000", 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.477e-06 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.522396+0000", 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00072250799999999996 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.522422+0000", 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5505999999999999e-05 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.522436+0000", 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4435e-05 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.616799+0000", 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094362980999999999 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.616812+0000", 2024-09-16T04:41:59.365 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3239e-05 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817885+0000", 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.201072954 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817965+0000", 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9869000000000004e-05 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 42/41 ECSubWriteReply(tid=33, last_complete=42'8, committed=1, applied=1))", 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.610676+0000", 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.6755295930000003, 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20704228299999999, 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610676+0000", 2024-09-16T04:41:59.367 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610676+0000", 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610677+0000", 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610680+0000", 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0699999999999998e-06 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610693+0000", 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3162e-05 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610700+0000", 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2969999999999998e-06 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817685+0000", 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206984848 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817718+0000", 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3315999999999997e-05 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.370 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 42/41 ECSubWriteReply(tid=29, last_complete=42'6, committed=1, applied=1))", 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.610878+0000", 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.6753273430000002, 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.207101543, 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610878+0000", 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.371 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610878+0000", 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610879+0000", 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4600000000000003e-07 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.372 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610884+0000", 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2139999999999998e-06 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610903+0000", 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610914+0000", 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1392999999999999e-05 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817821+0000", 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206906426 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817980+0000", 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000159419 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 44/41 ECSubWriteReply(tid=43, last_complete=44'19, committed=1, applied=1))", 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.614971+0000", 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.6712343890000003, 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21339023400000001, 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614971+0000", 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.375 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614971+0000", 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614974+0000", 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.12e-06 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614985+0000", 2024-09-16T04:41:59.376 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1293e-05 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615025+0000", 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0500999999999997e-05 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615047+0000", 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1239000000000001e-05 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.377 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828311+0000", 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.213264333 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828362+0000", 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0748000000000003e-05 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 44/41 ECSubWriteReply(tid=44, last_complete=44'20, committed=1, applied=1))", 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.615085+0000", 2024-09-16T04:41:59.378 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.6711205370000002, 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21334994199999999, 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615085+0000", 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615085+0000", 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615086+0000", 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.009e-06 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615092+0000", 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6529999999999999e-06 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615116+0000", 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3632999999999999e-05 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615130+0000", 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4759e-05 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828408+0000", 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.213277732 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828435+0000", 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7155999999999999e-05 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 44/41 ECSubWriteReply(tid=43, last_complete=44'19, committed=1, applied=1))", 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.726134+0000", 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.5600720690000003, 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.303476629, 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726134+0000", 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.383 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726134+0000", 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726135+0000", 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726166+0000", 2024-09-16T04:41:59.384 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1001999999999999e-05 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726182+0000", 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6846000000000001e-05 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726193+0000", 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1129e-05 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:41:59.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:55.029574+0000", 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30338042300000001 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:55.029610+0000", 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6439000000000001e-05 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:41:59.386 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:41:59.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:41:59.389 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:41:59.384+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:41:59.389 INFO:tasks.rados.rados.0.smithi165.stdout:142: finishing write tid 1 to smithi16530038-3 2024-09-16T04:41:59.489 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:41:59.483+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:41:59.491 INFO:tasks.rados.rados.0.smithi165.stdout:142: finishing write tid 2 to smithi16530038-3 2024-09-16T04:41:59.520 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:59.520 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:41:59.520 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:41:59.520 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:59.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:41:59.572 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 14 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T04:41:59.590 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:41:59.588+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:41:59.593 INFO:tasks.rados.rados.0.smithi165.stdout:144: finishing rollback tid 0 to smithi16530038-48 2024-09-16T04:41:59.593 INFO:tasks.rados.rados.0.smithi165.stdout:144: finishing rollback tid 1 to smithi16530038-48 2024-09-16T04:41:59.652 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:59.652 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:41:59.652 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:41:59.652 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:41:59.652 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:59.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:41:59.675 INFO:tasks.rados.rados.0.smithi165.stdout:144: finishing rollback tid 2 to smithi16530038-48 2024-09-16T04:41:59.675 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 56 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T04:41:59.690 INFO:tasks.rados.rados.0.smithi165.stdout:142: finishing write tid 3 to smithi16530038-3 2024-09-16T04:41:59.691 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:41:59.687+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:41:59.768 INFO:tasks.rados.rados.0.smithi165.stdout:143: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T04:41:59.791 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:41:59.787+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:41:59.805 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:41:59.805 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:41:59.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:41:59.805 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:41:59.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:122 3.as0 3:5619390f:::smithi16530038-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.123556781, 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51059284400000005, 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.806 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678088+0000", 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.12e-06 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678098+0000", 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0000000000000001e-05 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678119+0000", 2024-09-16T04:41:59.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1676e-05 2024-09-16T04:41:59.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678135+0000", 2024-09-16T04:41:59.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5526999999999999e-05 2024-09-16T04:41:59.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682759+0000", 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046242649999999998 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682887+0000", 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012783599999999999 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.683968+0000", 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010808090000000001 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.784795+0000", 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10082761899999999 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188432+0000", 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40363656399999998 2024-09-16T04:41:59.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188678+0000", 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024642799999999998 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:123 3.as0 3:5619390f:::smithi16530038-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.812 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.123425445000001, 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91283640799999999, 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 123 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.813 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678219+0000", 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4030000000000001e-06 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678231+0000", 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678267+0000", 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5719000000000001e-05 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678291+0000", 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4006e-05 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684378+0000", 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060877320000000002 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684496+0000", 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118076 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188804+0000", 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50430794400000001 2024-09-16T04:41:59.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.189032+0000", 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000227235 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.591053+0000", 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402021666 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:133 3.as0 3:5f4281cc:::smithi16530038-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.91823726, 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50755930599999999, 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.819 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883408+0000", 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8669999999999998e-06 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884235+0000", 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082694699999999997 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884270+0000", 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4690000000000002e-05 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884286+0000", 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6082999999999998e-05 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884441+0000", 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155852 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884838+0000", 2024-09-16T04:41:59.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000396494 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.886865+0000", 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020266709999999999 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965465+0000", 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078600544999999994 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390790+0000", 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42532448499999997 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390964+0000", 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017467200000000001 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:134 3.as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.91732577, 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50704782699999995, 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.825 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884318+0000", 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884326+0000", 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.32e-06 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884339+0000", 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.274e-05 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884348+0000", 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0229999999999998e-06 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888432+0000", 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040844899999999996 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888564+0000", 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131419 2024-09-16T04:41:59.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889204+0000", 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064011899999999997 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965469+0000", 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076265854999999994 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:59.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391186+0000", 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.425716382 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391364+0000", 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017849599999999999 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:135 3.as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.830 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.917231674, 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0075776860000001, 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884412+0000", 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3399999999999997e-07 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884416+0000", 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3059999999999999e-06 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884428+0000", 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2500000000000001e-05 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884439+0000", 2024-09-16T04:41:59.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0662999999999999e-05 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889599+0000", 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051597179999999998 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889722+0000", 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012330300000000001 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392670+0000", 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50294774399999997 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392905+0000", 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023530799999999999 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.891988+0000", 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49908321 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.835 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:171 3.as0 3:5619390f:::smithi16530038-41:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896540+0000", 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.905102633, 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40213990599999999, 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.836 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896540+0000", 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896540+0000", 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896542+0000", 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.554e-06 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896552+0000", 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.01e-05 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896583+0000", 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0423e-05 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896598+0000", 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5043e-05 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896717+0000", 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118836 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896906+0000", 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018926199999999999 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.897736+0000", 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082986699999999997 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.899917+0000", 2024-09-16T04:41:59.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021813000000000002 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.298570+0000", 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39865285499999997 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.298680+0000", 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109666 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.841 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:176 3.as0 3:5619390f:::smithi16530038-41:head [copy-from ver 10] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.726944895999999, 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71635610400000005, 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074701+0000", 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9500000000000003e-06 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074716+0000", 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4523e-05 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074742+0000", 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5795999999999998e-05 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074764+0000", 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2646999999999998e-05 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074937+0000", 2024-09-16T04:41:59.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017243599999999999 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074957+0000", 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9873000000000001e-05 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178498+0000", 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103540929 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178630+0000", 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132399 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.493151+0000", 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.3145212 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.597069+0000", 2024-09-16T04:41:59.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103917643 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.790921+0000", 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19385244300000001 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791054+0000", 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013226500000000001 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.848 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:177 3.as0 3:5619390f:::smithi16530038-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.726825363, 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42059460999999998, 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074819+0000", 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074828+0000", 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7399999999999993e-06 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074850+0000", 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2802999999999998e-05 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074873+0000", 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2180000000000001e-05 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075093+0000", 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000220645 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075109+0000", 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5823000000000001e-05 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178898+0000", 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10378908100000001 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178962+0000", 2024-09-16T04:41:59.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489000000000001e-05 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495097+0000", 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31613566599999998 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495230+0000", 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133039 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495412+0000", 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018130800000000001 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:194 3.as0 3:5f4281cc:::smithi16530038-45:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.701666331, 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59800688199999996, 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.855 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 194 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099979+0000", 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6400000000000001e-06 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099990+0000", 2024-09-16T04:41:59.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.15e-05 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100022+0000", 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1569999999999998e-05 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100039+0000", 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7486000000000002e-05 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179005+0000", 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078965628999999996 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179189+0000", 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184155 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.486772+0000", 2024-09-16T04:41:59.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30758345500000001 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495583+0000", 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088103490000000003 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697796+0000", 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20221325300000001 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697983+0000", 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018684499999999999 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:276 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.7028628470000005, 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71405619499999995, 2024-09-16T04:41:59.861 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 276 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098783+0000", 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098798+0000", 2024-09-16T04:41:59.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5213e-05 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098845+0000", 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6853e-05 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098872+0000", 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6622999999999999e-05 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099054+0000", 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018249499999999999 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099075+0000", 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.065e-05 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207663+0000", 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858786400000001 2024-09-16T04:41:59.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207828+0000", 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016534199999999999 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812836+0000", 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60500746100000002 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:277 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.702714995, 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81467523799999997, 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 277 2024-09-16T04:41:59.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098929+0000", 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4670000000000001e-06 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098938+0000", 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098964+0000", 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5979999999999999e-05 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098980+0000", 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5917e-05 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099168+0000", 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000188228 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099176+0000", 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.307912+0000", 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20873598700000001 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.308068+0000", 2024-09-16T04:41:59.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156448 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913603+0000", 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60553407699999995 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:278 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:41:59.872 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.7026388849999998, 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81515018400000006, 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 278 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099005+0000", 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-06 2024-09-16T04:41:59.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099012+0000", 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9e-06 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099031+0000", 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9303000000000001e-05 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099046+0000", 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5003e-05 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099213+0000", 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000166773 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:41:59.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099218+0000", 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408464+0000", 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30924607700000001 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408612+0000", 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014793900000000001 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.914154+0000", 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50554194200000002 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.309194+0000", 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.492448445, 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40233833499999999, 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.878 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309194+0000", 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309194+0000", 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309196+0000", 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1260000000000001e-06 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309232+0000", 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6443999999999999e-05 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309253+0000", 2024-09-16T04:41:59.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0599999999999999e-05 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309265+0000", 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2099999999999999e-05 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.711464+0000", 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402199313 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.711532+0000", 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7751999999999998e-05 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.as0 43/32 ECSubRead(tid=39, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.3927220560000002, 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40299000499999998, 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.882 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408986+0000", 2024-09-16T04:41:59.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409043+0000", 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6589999999999999e-05 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811767+0000", 2024-09-16T04:41:59.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40272446200000001 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811910+0000", 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014330300000000001 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=37, attrs_read=0))", 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:41:59.886 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.3923227349999996, 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.403543547, 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:41:59.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409389+0000", 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9126000000000002e-05 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409407+0000", 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8097e-05 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812374+0000", 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40296689699999999 2024-09-16T04:41:59.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812863+0000", 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048942699999999996 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.3919622780000003, 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40266660700000001, 2024-09-16T04:41:59.890 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409681+0000", 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2699999999999998e-07 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409710+0000", 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9036000000000001e-05 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409721+0000", 2024-09-16T04:41:59.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1187e-05 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409729+0000", 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7260000000000002e-06 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812298+0000", 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40256918800000002 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812347+0000", 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8542999999999999e-05 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.3918861790000001, 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402371019, 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.894 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409757+0000", 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409802+0000", 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4969999999999998e-05 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409813+0000", 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1127e-05 2024-09-16T04:41:59.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409820+0000", 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4499999999999998e-06 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812023+0000", 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40220215599999998 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812127+0000", 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104549 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.3917194999999998, 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40309429400000002, 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.898 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409924+0000", 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409963+0000", 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8753e-05 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409981+0000", 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8737e-05 2024-09-16T04:41:59.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409991+0000", 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3889999999999994e-06 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812922+0000", 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40293080100000001 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.813017+0000", 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5457000000000004e-05 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.410028+0000", 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.3916139640000003, 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40223944, 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.902 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410028+0000", 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410028+0000", 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410029+0000", 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410061+0000", 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2246000000000003e-05 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410076+0000", 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.504e-05 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.510190+0000", 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100113808 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812211+0000", 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30202088300000002 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812268+0000", 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6765999999999998e-05 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.2906392330000003, 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40262853199999998, 2024-09-16T04:41:59.906 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511049+0000", 2024-09-16T04:41:59.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5912999999999999e-05 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511068+0000", 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.855e-05 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913200+0000", 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40213229900000003 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:41:59.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913632+0000", 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043176999999999999 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:41:59.910 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:41:59.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:41:59.912 INFO:tasks.rados.rados.0.smithi165.stdout:125: done (5 left) 2024-09-16T04:41:59.912 INFO:tasks.rados.rados.0.smithi165.stdout:141: done (4 left) 2024-09-16T04:41:59.912 INFO:tasks.rados.rados.0.smithi165.stdout:143: done (3 left) 2024-09-16T04:41:59.912 INFO:tasks.rados.rados.0.smithi165.stdout:144: done (2 left) 2024-09-16T04:41:59.912 INFO:tasks.rados.rados.0.smithi165.stdout:145: done (1 left) 2024-09-16T04:41:59.912 INFO:tasks.rados.rados.0.smithi165.stdout:146: delete oid 36 current snap is 5 2024-09-16T04:41:59.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:41:59.914 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:41:59.908+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:41:59.997 INFO:tasks.rados.rados.0.smithi165.stdout:142: finishing write tid 4 to smithi16530038-3 2024-09-16T04:41:59.997 INFO:tasks.rados.rados.0.smithi165.stdout:142: oid 3 updating version 0 to 26 2024-09-16T04:41:59.997 INFO:tasks.rados.rados.0.smithi165.stdout:142: oid 3 updating version 26 to 28 2024-09-16T04:41:59.997 INFO:tasks.rados.rados.0.smithi165.stdout:142: oid 3 version 28 is already newer than 27 2024-09-16T04:41:59.997 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 28 (ObjNum 70 snap 5 seq_num 70) dirty exists 2024-09-16T04:41:59.997 INFO:tasks.rados.rados.0.smithi165.stdout:142: left oid 3 (ObjNum 70 snap 5 seq_num 70) 2024-09-16T04:42:00.013 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:00.007+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:00.073 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:00.085 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:00.086 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:00.094 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:00.094 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:00.098 INFO:tasks.rados.rados.0.smithi165.stdout:142: done (1 left) 2024-09-16T04:42:00.098 INFO:tasks.rados.rados.0.smithi165.stdout:146: done (0 left) 2024-09-16T04:42:00.099 INFO:tasks.rados.rados.0.smithi165.stdout:147: copy_from oid 36 from oid 1 current snap is 5 2024-09-16T04:42:00.099 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 31 current snap is 5 2024-09-16T04:42:00.099 INFO:tasks.rados.rados.0.smithi165.stdout:148: seq_num 73 ranges {0=458752,458752=475136,933888=294912} 2024-09-16T04:42:00.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:00.106 INFO:tasks.rados.rados.0.smithi165.stdout:148: writing smithi16530038-31 from 0 to 458752 tid 1 2024-09-16T04:42:00.113 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:00.112+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:00.115 INFO:tasks.rados.rados.0.smithi165.stdout:148: writing smithi16530038-31 from 458752 to 933888 tid 2 2024-09-16T04:42:00.119 INFO:tasks.rados.rados.0.smithi165.stdout:148: writing smithi16530038-31 from 933888 to 1228800 tid 3 2024-09-16T04:42:00.119 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 8 current snap is 5 2024-09-16T04:42:00.119 INFO:tasks.rados.rados.0.smithi165.stdout:149: seq_num 74 ranges {131072=475136,606208=622592,1228800=147456} 2024-09-16T04:42:00.120 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:00.121 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:00.129 INFO:tasks.rados.rados.0.smithi165.stdout:149: writing smithi16530038-8 from 131072 to 606208 tid 1 2024-09-16T04:42:00.138 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:00.139 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:00.139 INFO:tasks.rados.rados.0.smithi165.stdout:149: writing smithi16530038-8 from 606208 to 1228800 tid 2 2024-09-16T04:42:00.142 INFO:tasks.rados.rados.0.smithi165.stdout:149: writing smithi16530038-8 from 1228800 to 1376256 tid 3 2024-09-16T04:42:00.142 INFO:tasks.rados.rados.0.smithi165.stdout:150: copy_from oid 41 from oid 17 current snap is 5 2024-09-16T04:42:00.142 INFO:tasks.rados.rados.0.smithi165.stdout:151: copy_from oid 14 from oid 25 current snap is 5 2024-09-16T04:42:00.142 INFO:tasks.rados.rados.0.smithi165.stdout:152: snap_create 2024-09-16T04:42:00.151 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:42:00.155 INFO:tasks.rados.rados.0.smithi165.stdout:152: done (5 left) 2024-09-16T04:42:00.155 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 16 current snap is 6 2024-09-16T04:42:00.155 INFO:tasks.rados.rados.0.smithi165.stdout:153: seq_num 75 ranges {262144=491520,753664=770048} 2024-09-16T04:42:00.166 INFO:tasks.rados.rados.0.smithi165.stdout:153: writing smithi16530038-16 from 262144 to 753664 tid 1 2024-09-16T04:42:00.178 INFO:tasks.rados.rados.0.smithi165.stdout:153: writing smithi16530038-16 from 753664 to 1523712 tid 2 2024-09-16T04:42:00.178 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 18 current snap is 6 2024-09-16T04:42:00.178 INFO:tasks.rados.rados.0.smithi165.stdout:154: seq_num 76 ranges {0=507904,507904=507904,1015808=262144} 2024-09-16T04:42:00.185 INFO:tasks.rados.rados.0.smithi165.stdout:154: writing smithi16530038-18 from 0 to 507904 tid 1 2024-09-16T04:42:00.192 INFO:tasks.rados.rados.0.smithi165.stdout:154: writing smithi16530038-18 from 507904 to 1015808 tid 2 2024-09-16T04:42:00.196 INFO:tasks.rados.rados.0.smithi165.stdout:154: writing smithi16530038-18 from 1015808 to 1277952 tid 3 2024-09-16T04:42:00.196 INFO:tasks.rados.rados.0.smithi165.stdout:155: rollback oid 27 current snap is 6 2024-09-16T04:42:00.196 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 27 to 1 2024-09-16T04:42:00.196 INFO:tasks.rados.rados.0.smithi165.stdout:156: snap_create 2024-09-16T04:42:00.221 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:00.220+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:00.226 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:00.226 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:00.226 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:00.226 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:00.226 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:00.227 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:00.227 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:00.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:00.250 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:00.250 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:00.274 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:00.275 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:00.300 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:00.301 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:00.315 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:00.312+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:00.325 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:00.325 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:00.350 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:00.350 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:00.365 INFO:tasks.rados.rados.0.smithi165.stdout:151: finishing copy_from racing read to smithi16530038-14 2024-09-16T04:42:00.411 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:00.411 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:00.417 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:00.412+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:00.425 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:00.426 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:00.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:00.453 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:00.482 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:00.482 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:00.512 INFO:tasks.rados.rados.0.smithi165.stdout:151: finishing copy_from to smithi16530038-14 2024-09-16T04:42:00.512 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 8 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T04:42:00.514 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:00.514 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:00.514 INFO:tasks.rados.rados.0.smithi165.stdout:149: finishing write tid 1 to smithi16530038-8 2024-09-16T04:42:00.517 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:00.515+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:00.520 INFO:tasks.rados.rados.0.smithi165.stdout:147: finishing copy_from racing read to smithi16530038-36 2024-09-16T04:42:00.553 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:00.553 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:00.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:00.595 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:00.595 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:00.612 INFO:tasks.rados.rados.0.smithi165.stdout:148: finishing write tid 1 to smithi16530038-31 2024-09-16T04:42:00.613 INFO:tasks.rados.rados.0.smithi165.stdout:148: finishing write tid 2 to smithi16530038-31 2024-09-16T04:42:00.614 INFO:tasks.rados.rados.0.smithi165.stdout:153: finishing write tid 1 to smithi16530038-16 2024-09-16T04:42:00.619 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:00.616+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:00.620 INFO:tasks.rados.rados.0.smithi165.stdout:149: finishing write tid 2 to smithi16530038-8 2024-09-16T04:42:00.665 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing rollback tid 0 to smithi16530038-27 2024-09-16T04:42:00.666 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing rollback tid 1 to smithi16530038-27 2024-09-16T04:42:00.666 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing rollback tid 2 to smithi16530038-27 2024-09-16T04:42:00.666 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 30 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-16T04:42:00.699 INFO:tasks.rados.rados.0.smithi165.stdout:154: finishing write tid 1 to smithi16530038-18 2024-09-16T04:42:00.720 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:00.715+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:00.721 INFO:tasks.rados.rados.0.smithi165.stdout:149: finishing write tid 3 to smithi16530038-8 2024-09-16T04:42:00.732 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:00.733 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:00.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:00.749 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:00.749 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:00.783 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:42:00.783 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:42:00.783 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T04:42:00.783 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:00.799 INFO:tasks.rados.rados.0.smithi165.stdout:154: finishing write tid 2 to smithi16530038-18 2024-09-16T04:42:00.800 INFO:tasks.rados.rados.0.smithi165.stdout:154: finishing write tid 3 to smithi16530038-18 2024-09-16T04:42:00.813 INFO:tasks.rados.rados.0.smithi165.stdout:148: finishing write tid 3 to smithi16530038-31 2024-09-16T04:42:00.813 INFO:tasks.rados.rados.0.smithi165.stdout:148: finishing write tid 4 to smithi16530038-31 2024-09-16T04:42:00.821 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:00.816+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:00.821 INFO:tasks.rados.rados.0.smithi165.stdout:147: finishing copy_from to smithi16530038-36 2024-09-16T04:42:00.821 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:00.913 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:00.914 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:00.915 INFO:tasks.rados.rados.0.smithi165.stdout:153: finishing write tid 2 to smithi16530038-16 2024-09-16T04:42:00.917 INFO:tasks.rados.rados.0.smithi165.stdout:148: finishing write tid 5 to smithi16530038-31 2024-09-16T04:42:00.917 INFO:tasks.rados.rados.0.smithi165.stdout:148: oid 31 updating version 0 to 17 2024-09-16T04:42:00.918 INFO:tasks.rados.rados.0.smithi165.stdout:148: oid 31 version 17 is already newer than 16 2024-09-16T04:42:00.918 INFO:tasks.rados.rados.0.smithi165.stdout:148: oid 31 updating version 17 to 18 2024-09-16T04:42:00.918 INFO:tasks.rados.rados.0.smithi165.stdout:148: oid 31 version 18 is already newer than 15 2024-09-16T04:42:00.918 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 18 (ObjNum 73 snap 5 seq_num 73) dirty exists 2024-09-16T04:42:00.918 INFO:tasks.rados.rados.0.smithi165.stdout:148: left oid 31 (ObjNum 73 snap 5 seq_num 73) 2024-09-16T04:42:00.922 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:00.920+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:00.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:42:01.003 INFO:tasks.rados.rados.0.smithi165.stdout:154: finishing write tid 4 to smithi16530038-18 2024-09-16T04:42:01.004 INFO:tasks.rados.rados.0.smithi165.stdout:153: finishing write tid 3 to smithi16530038-16 2024-09-16T04:42:01.023 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:01.019+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:01.028 INFO:tasks.rados.rados.0.smithi165.stdout:149: finishing write tid 4 to smithi16530038-8 2024-09-16T04:42:01.029 INFO:tasks.rados.rados.0.smithi165.stdout:150: finishing copy_from racing read to smithi16530038-41 2024-09-16T04:42:01.092 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:01.093 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:01.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:42:01.124 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:01.119+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:01.140 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:01.141 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:42:01.158 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-09-16T04:42:00.997850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37667379,"num_objects":75,"num_object_clones":21,"num_object_copies":505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":3,"num_read":308,"num_read_kb":20627,"num_write":388,"num_write_kb":26934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":457,"ondisk_log_size":457,"up":193,"acting":193,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":526656,"kb_used_data":69152,"kb_used_omap":24,"kb_used_meta":457063,"kb_avail":1509422784,"statfs":{"total":1546188226560,"available":1545648930816,"internally_reserved":0,"allocated":70811648,"data_stored":87786174,"data_compressed":1300420,"data_compressed_allocated":1818624,"data_compressed_original":4497408,"omap_allocated":25522,"internal_metadata":468032590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,0,0,1,2,4,1,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6234112,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":3,"num_read":9,"num_read_kb":6816,"num_write":24,"num_write_kb":-5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-1523712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001507"},"pg_stats":[{"pgid":"3.18","version":"35'2","reported_seq":54,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-16T04:41:59.061921+0000","last_change":"2024-09-16T04:41:59.154867+0000","last_active":"2024-09-16T04:41:59.154867+0000","last_peered":"2024-09-16T04:41:59.061921+0000","last_clean":"2024-09-16T04:41:59.061921+0000","last_became_active":"2024-09-16T04:41:52.415756+0000","last_became_peered":"2024-09-16T04:41:52.415756+0000","last_unstale":"2024-09-16T04:41:59.061921+0000","last_undegraded":"2024-09-16T04:41:59.061921+0000","last_fullsized":"2024-09-16T04:41:59.061921+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:59.061655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.061655+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0.0024114679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2,6,14,15],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.17","version":"44'6","reported_seq":113,"reported_epoch":49,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:41:59.368511+0000","last_change":"2024-09-16T04:41:58.891324+0000","last_active":"2024-09-16T04:41:59.368511+0000","last_peered":"2024-09-16T04:41:59.368511+0000","last_clean":"2024-09-16T04:41:59.368511+0000","last_became_active":"2024-09-16T04:41:52.612293+0000","last_became_peered":"2024-09-16T04:41:52.612293+0000","last_unstale":"2024-09-16T04:41:59.368511+0000","last_undegraded":"2024-09-16T04:41:59.368511+0000","last_fullsized":"2024-09-16T04:41:59.368511+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078374599999999999,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1777,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.16","version":"35'2","reported_seq":78,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:41:57.146355+0000","last_change":"2024-09-16T04:41:57.147172+0000","last_active":"2024-09-16T04:41:57.146355+0000","last_peered":"2024-09-16T04:41:57.146355+0000","last_clean":"2024-09-16T04:41:57.146355+0000","last_became_active":"2024-09-16T04:41:52.305704+0000","last_became_peered":"2024-09-16T04:41:52.305704+0000","last_unstale":"2024-09-16T04:41:57.146355+0000","last_undegraded":"2024-09-16T04:41:57.146355+0000","last_fullsized":"2024-09-16T04:41:57.146355+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008901,"stat_sum":{"num_bytes":40960,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":242,"num_write":1,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.15","version":"36'6","reported_seq":36,"reported_epoch":49,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T04:41:59.167241+0000","last_change":"2024-09-16T04:41:53.449148+0000","last_active":"2024-09-16T04:41:59.167241+0000","last_peered":"2024-09-16T04:41:59.167241+0000","last_clean":"2024-09-16T04:41:46.181856+0000","last_became_active":"2024-09-16T04:41:53.445446+0000","last_became_peered":"2024-09-16T04:41:53.445446+0000","last_unstale":"2024-09-16T04:41:59.167241+0000","last_undegraded":"2024-09-16T04:41:59.167241+0000","last_fullsized":"2024-09-16T04:41:59.167241+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":761856,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,14,0,6],"avail_no_missing":["15(0)","0(5)","2(5)","3(2)","4(2)","6(1)","6(6)","7(4)","8(1)","9(0)","13(3)","14(4)","15(6)"],"object_location_counts":[{"shards":"0(5),4(2),6(6),8(1),13(3),14(4),15(0)","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.14","version":"49'28","reported_seq":122,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.389789+0000","last_change":"2024-09-16T04:41:59.184077+0000","last_active":"2024-09-16T04:41:59.389789+0000","last_peered":"2024-09-16T04:41:59.389789+0000","last_clean":"2024-09-16T04:41:59.389789+0000","last_became_active":"2024-09-16T04:41:52.409152+0000","last_became_peered":"2024-09-16T04:41:52.409152+0000","last_unstale":"2024-09-16T04:41:59.389789+0000","last_undegraded":"2024-09-16T04:41:59.389789+0000","last_fullsized":"2024-09-16T04:41:59.389789+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:25.382353+0000","scrub_duration":899,"objects_trimmed":0,"snaptrim_duration":0.0055386970000000004,"stat_sum":{"num_bytes":3555328,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.253184+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:41:59.253184+0000","last_peered":"2024-09-16T04:41:59.253184+0000","last_clean":"2024-09-16T04:41:59.253184+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:41:59.253184+0000","last_undegraded":"2024-09-16T04:41:59.253184+0000","last_fullsized":"2024-09-16T04:41:59.253184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":84,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:41:57.146938+0000","last_change":"2024-09-16T04:41:57.247663+0000","last_active":"2024-09-16T04:41:57.146938+0000","last_peered":"2024-09-16T04:41:57.146938+0000","last_clean":"2024-09-16T04:41:57.146938+0000","last_became_active":"2024-09-16T04:41:48.483004+0000","last_became_peered":"2024-09-16T04:41:48.483004+0000","last_unstale":"2024-09-16T04:41:57.146938+0000","last_undegraded":"2024-09-16T04:41:57.146938+0000","last_fullsized":"2024-09-16T04:41:57.146938+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:07.014267+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.10118664199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.178606+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:41:59.178606+0000","last_peered":"2024-09-16T04:41:59.178606+0000","last_clean":"2024-09-16T04:41:59.178606+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:41:59.178606+0000","last_undegraded":"2024-09-16T04:41:59.178606+0000","last_fullsized":"2024-09-16T04:41:59.178606+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":89,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.287892+0000","last_change":"2024-09-16T04:41:59.289294+0000","last_active":"2024-09-16T04:41:59.287892+0000","last_peered":"2024-09-16T04:41:59.287892+0000","last_clean":"2024-09-16T04:41:59.287892+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:41:59.287892+0000","last_undegraded":"2024-09-16T04:41:59.287892+0000","last_fullsized":"2024-09-16T04:41:59.287892+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003216081,"stat_sum":{"num_bytes":16384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.359941+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:41:59.359941+0000","last_peered":"2024-09-16T04:41:59.359941+0000","last_clean":"2024-09-16T04:41:59.359941+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:41:59.359941+0000","last_undegraded":"2024-09-16T04:41:59.359941+0000","last_fullsized":"2024-09-16T04:41:59.359941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"36'10","reported_seq":105,"reported_epoch":49,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:41:59.179089+0000","last_change":"2024-09-16T04:41:58.656470+0000","last_active":"2024-09-16T04:41:59.179089+0000","last_peered":"2024-09-16T04:41:59.179089+0000","last_clean":"2024-09-16T04:41:59.179089+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:41:59.179089+0000","last_undegraded":"2024-09-16T04:41:59.179089+0000","last_fullsized":"2024-09-16T04:41:59.179089+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074566699999999999,"stat_sum":{"num_bytes":737280,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":60,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.287958+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:41:59.287958+0000","last_peered":"2024-09-16T04:41:59.287958+0000","last_clean":"2024-09-16T04:41:59.287958+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:41:59.287958+0000","last_undegraded":"2024-09-16T04:41:59.287958+0000","last_fullsized":"2024-09-16T04:41:59.287958+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"47'15","reported_seq":121,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.269248+0000","last_change":"2024-09-16T04:41:59.269998+0000","last_active":"2024-09-16T04:41:59.269248+0000","last_peered":"2024-09-16T04:41:59.269248+0000","last_clean":"2024-09-16T04:41:59.269248+0000","last_became_active":"2024-09-16T04:41:48.475648+0000","last_became_peered":"2024-09-16T04:41:48.475648+0000","last_unstale":"2024-09-16T04:41:59.269248+0000","last_undegraded":"2024-09-16T04:41:59.269248+0000","last_fullsized":"2024-09-16T04:41:59.269248+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:12.300870+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.001245527,"stat_sum":{"num_bytes":3637248,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":674,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.3","version":"47'9","reported_seq":108,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:42:00.411162+0000","last_change":"2024-09-16T04:41:59.172465+0000","last_active":"2024-09-16T04:42:00.411162+0000","last_peered":"2024-09-16T04:42:00.411162+0000","last_clean":"2024-09-16T04:42:00.411162+0000","last_became_active":"2024-09-16T04:41:48.281236+0000","last_became_peered":"2024-09-16T04:41:48.281236+0000","last_unstale":"2024-09-16T04:42:00.411162+0000","last_undegraded":"2024-09-16T04:42:00.411162+0000","last_fullsized":"2024-09-16T04:42:00.411162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:41:49.293927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.293927+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:50.152777+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.0024804720000000001,"stat_sum":{"num_bytes":1933312,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":1072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.2","version":"28'2","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.476121+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:41:59.476121+0000","last_peered":"2024-09-16T04:41:59.476121+0000","last_clean":"2024-09-16T04:41:59.476121+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:41:59.476121+0000","last_undegraded":"2024-09-16T04:41:59.476121+0000","last_fullsized":"2024-09-16T04:41:59.476121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":99,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.273721+0000","last_change":"2024-09-16T04:41:59.275435+0000","last_active":"2024-09-16T04:41:59.273721+0000","last_peered":"2024-09-16T04:41:59.273721+0000","last_clean":"2024-09-16T04:41:59.273721+0000","last_became_active":"2024-09-16T04:41:48.383143+0000","last_became_peered":"2024-09-16T04:41:48.383143+0000","last_unstale":"2024-09-16T04:41:59.273721+0000","last_undegraded":"2024-09-16T04:41:59.273721+0000","last_fullsized":"2024-09-16T04:41:59.273721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:49.599894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.599894+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:59.526444+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0.00234228,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:42:00.404244+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:42:00.404244+0000","last_peered":"2024-09-16T04:42:00.404244+0000","last_clean":"2024-09-16T04:42:00.404244+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:42:00.404244+0000","last_undegraded":"2024-09-16T04:42:00.404244+0000","last_fullsized":"2024-09-16T04:42:00.404244+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"44'14","reported_seq":116,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.287248+0000","last_change":"2024-09-16T04:41:59.289706+0000","last_active":"2024-09-16T04:41:59.287248+0000","last_peered":"2024-09-16T04:41:59.287248+0000","last_clean":"2024-09-16T04:41:59.287248+0000","last_became_active":"2024-09-16T04:41:48.404175+0000","last_became_peered":"2024-09-16T04:41:48.404175+0000","last_unstale":"2024-09-16T04:41:59.287248+0000","last_undegraded":"2024-09-16T04:41:59.287248+0000","last_fullsized":"2024-09-16T04:41:59.287248+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:50.183380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.183380+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:54.720954+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0.0041603789999999996,"stat_sum":{"num_bytes":1376256,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.275594+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:41:59.275594+0000","last_peered":"2024-09-16T04:41:59.275594+0000","last_clean":"2024-09-16T04:41:59.275594+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:41:59.275594+0000","last_undegraded":"2024-09-16T04:41:59.275594+0000","last_fullsized":"2024-09-16T04:41:59.275594+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"36'4","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:42:00.310590+0000","last_change":"2024-09-16T04:41:59.171135+0000","last_active":"2024-09-16T04:42:00.310590+0000","last_peered":"2024-09-16T04:42:00.310590+0000","last_clean":"2024-09-16T04:42:00.310590+0000","last_became_active":"2024-09-16T04:41:48.477867+0000","last_became_peered":"2024-09-16T04:41:48.477867+0000","last_unstale":"2024-09-16T04:42:00.310590+0000","last_undegraded":"2024-09-16T04:42:00.310590+0000","last_fullsized":"2024-09-16T04:42:00.310590+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.564481+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:56.648830+0000","scrub_duration":1083,"objects_trimmed":0,"snaptrim_duration":0.0016201869999999999,"stat_sum":{"num_bytes":720896,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.3","version":"26'1","reported_seq":62,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.254010+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:41:59.254010+0000","last_peered":"2024-09-16T04:41:59.254010+0000","last_clean":"2024-09-16T04:41:59.254010+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:41:59.254010+0000","last_undegraded":"2024-09-16T04:41:59.254010+0000","last_fullsized":"2024-09-16T04:41:59.254010+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":257,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:42:00.310328+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:42:00.310328+0000","last_peered":"2024-09-16T04:42:00.310328+0000","last_clean":"2024-09-16T04:42:00.310328+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:42:00.310328+0000","last_undegraded":"2024-09-16T04:42:00.310328+0000","last_fullsized":"2024-09-16T04:42:00.310328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"44'8","reported_seq":100,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-16T04:41:58.166630+0000","last_change":"2024-09-16T04:41:59.158056+0000","last_active":"2024-09-16T04:41:59.158056+0000","last_peered":"2024-09-16T04:41:58.166630+0000","last_clean":"2024-09-16T04:41:58.166630+0000","last_became_active":"2024-09-16T04:41:48.481646+0000","last_became_peered":"2024-09-16T04:41:48.481646+0000","last_unstale":"2024-09-16T04:41:58.166630+0000","last_undegraded":"2024-09-16T04:41:58.166630+0000","last_fullsized":"2024-09-16T04:41:58.166630+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'4","last_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.115829+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0.00329167,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2257,"num_write":1,"num_write_kb":752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2,6,14,15],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.9","version":"48'26","reported_seq":126,"reported_epoch":49,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:41:59.272719+0000","last_change":"2024-09-16T04:41:58.808542+0000","last_active":"2024-09-16T04:41:59.272719+0000","last_peered":"2024-09-16T04:41:59.272719+0000","last_clean":"2024-09-16T04:41:59.272719+0000","last_became_active":"2024-09-16T04:41:58.352078+0000","last_became_peered":"2024-09-16T04:41:58.352078+0000","last_unstale":"2024-09-16T04:41:59.272719+0000","last_undegraded":"2024-09-16T04:41:59.272719+0000","last_fullsized":"2024-09-16T04:41:59.272719+0000","mapping_epoch":32,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'11","last_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:41:51.621244+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1979,"objects_trimmed":0,"snaptrim_duration":0.104460102,"stat_sum":{"num_bytes":2768896,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":197,"num_write":14,"num_write_kb":2704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.a","version":"48'17","reported_seq":114,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.276485+0000","last_change":"2024-09-16T04:41:59.277142+0000","last_active":"2024-09-16T04:41:59.276485+0000","last_peered":"2024-09-16T04:41:59.276485+0000","last_clean":"2024-09-16T04:41:59.276485+0000","last_became_active":"2024-09-16T04:41:42.059056+0000","last_became_peered":"2024-09-16T04:41:42.059056+0000","last_unstale":"2024-09-16T04:41:59.276485+0000","last_undegraded":"2024-09-16T04:41:59.276485+0000","last_fullsized":"2024-09-16T04:41:59.276485+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001071825,"stat_sum":{"num_bytes":1015808,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":3273,"num_write":32,"num_write_kb":2400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.b","version":"44'12","reported_seq":99,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.287453+0000","last_change":"2024-09-16T04:41:59.287790+0000","last_active":"2024-09-16T04:41:59.287453+0000","last_peered":"2024-09-16T04:41:59.287453+0000","last_clean":"2024-09-16T04:41:59.287453+0000","last_became_active":"2024-09-16T04:41:42.059548+0000","last_became_peered":"2024-09-16T04:41:42.059548+0000","last_unstale":"2024-09-16T04:41:59.287453+0000","last_undegraded":"2024-09-16T04:41:59.287453+0000","last_fullsized":"2024-09-16T04:41:59.287453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:57.002991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021014800000000002,"stat_sum":{"num_bytes":1818624,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":1927,"num_write":9,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.c","version":"45'23","reported_seq":95,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:41:57.265735+0000","last_change":"2024-09-16T04:41:57.266413+0000","last_active":"2024-09-16T04:41:57.265735+0000","last_peered":"2024-09-16T04:41:57.265735+0000","last_clean":"2024-09-16T04:41:57.265735+0000","last_became_active":"2024-09-16T04:41:42.057674+0000","last_became_peered":"2024-09-16T04:41:42.057674+0000","last_unstale":"2024-09-16T04:41:57.265735+0000","last_undegraded":"2024-09-16T04:41:57.265735+0000","last_fullsized":"2024-09-16T04:41:57.265735+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:43.616767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001017766,"stat_sum":{"num_bytes":2359296,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":1543,"num_write":54,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.d","version":"48'53","reported_seq":139,"reported_epoch":49,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:41:59.285356+0000","last_change":"2024-09-16T04:41:59.285414+0000","last_active":"2024-09-16T04:41:59.285356+0000","last_peered":"2024-09-16T04:41:59.285356+0000","last_clean":"2024-09-16T04:41:59.285356+0000","last_became_active":"2024-09-16T04:41:42.053696+0000","last_became_peered":"2024-09-16T04:41:42.053696+0000","last_unstale":"2024-09-16T04:41:59.285356+0000","last_undegraded":"2024-09-16T04:41:59.285356+0000","last_fullsized":"2024-09-16T04:41:59.285356+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:47.814606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001341863,"stat_sum":{"num_bytes":6488064,"num_objects":12,"num_object_clones":5,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":27,"num_read_kb":2795,"num_write":33,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.e","version":"45'12","reported_seq":100,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.160961+0000","last_change":"2024-09-16T04:41:59.161839+0000","last_active":"2024-09-16T04:41:59.160961+0000","last_peered":"2024-09-16T04:41:59.160961+0000","last_clean":"2024-09-16T04:41:59.160961+0000","last_became_active":"2024-09-16T04:41:42.158174+0000","last_became_peered":"2024-09-16T04:41:42.158174+0000","last_unstale":"2024-09-16T04:41:59.160961+0000","last_undegraded":"2024-09-16T04:41:59.160961+0000","last_fullsized":"2024-09-16T04:41:59.160961+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:59.900834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023586499999999999,"stat_sum":{"num_bytes":1212416,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":4087,"num_write":10,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.f","version":"44'17","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T04:41:57.166530+0000","last_change":"2024-09-16T04:41:57.260941+0000","last_active":"2024-09-16T04:41:57.166530+0000","last_peered":"2024-09-16T04:41:57.166530+0000","last_clean":"2024-09-16T04:41:57.166530+0000","last_became_active":"2024-09-16T04:41:41.950892+0000","last_became_peered":"2024-09-16T04:41:41.950892+0000","last_unstale":"2024-09-16T04:41:57.166530+0000","last_undegraded":"2024-09-16T04:41:57.166530+0000","last_fullsized":"2024-09-16T04:41:57.166530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:52.078948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.094825891999999995,"stat_sum":{"num_bytes":786432,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":1734,"num_write":10,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"2"}]},{"pgid":"3.10","version":"44'10","reported_seq":103,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.374100+0000","last_change":"2024-09-16T04:41:59.374657+0000","last_active":"2024-09-16T04:41:59.374100+0000","last_peered":"2024-09-16T04:41:59.374100+0000","last_clean":"2024-09-16T04:41:59.374100+0000","last_became_active":"2024-09-16T04:41:52.406889+0000","last_became_peered":"2024-09-16T04:41:52.406889+0000","last_unstale":"2024-09-16T04:41:59.374100+0000","last_undegraded":"2024-09-16T04:41:59.374100+0000","last_fullsized":"2024-09-16T04:41:59.374100+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:41:59.373567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.373567+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:14.936272+0000","scrub_duration":1467,"objects_trimmed":0,"snaptrim_duration":0.20085694300000001,"stat_sum":{"num_bytes":1310720,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.11","version":"47'24","reported_seq":125,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T04:41:59.285395+0000","last_change":"2024-09-16T04:41:59.286052+0000","last_active":"2024-09-16T04:41:59.285395+0000","last_peered":"2024-09-16T04:41:59.285395+0000","last_clean":"2024-09-16T04:41:59.285395+0000","last_became_active":"2024-09-16T04:41:52.407015+0000","last_became_peered":"2024-09-16T04:41:52.407015+0000","last_unstale":"2024-09-16T04:41:59.285395+0000","last_undegraded":"2024-09-16T04:41:59.285395+0000","last_fullsized":"2024-09-16T04:41:59.285395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'24","last_scrub_stamp":"2024-09-16T04:41:58.297098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.297098+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:15.123764+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.000850891,"stat_sum":{"num_bytes":3653632,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.12","version":"44'10","reported_seq":133,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:42:00.310591+0000","last_change":"2024-09-16T04:41:59.370900+0000","last_active":"2024-09-16T04:42:00.310591+0000","last_peered":"2024-09-16T04:42:00.310591+0000","last_clean":"2024-09-16T04:42:00.310591+0000","last_became_active":"2024-09-16T04:41:52.721423+0000","last_became_peered":"2024-09-16T04:41:52.721423+0000","last_unstale":"2024-09-16T04:42:00.310591+0000","last_undegraded":"2024-09-16T04:42:00.310591+0000","last_fullsized":"2024-09-16T04:42:00.310591+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:41:58.453618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.453618+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.397078+0000","scrub_duration":1119,"objects_trimmed":0,"snaptrim_duration":0.10140743100000001,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.13","version":"36'3","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:42:00.306824+0000","last_change":"2024-09-16T04:41:59.171359+0000","last_active":"2024-09-16T04:42:00.306824+0000","last_peered":"2024-09-16T04:42:00.306824+0000","last_clean":"2024-09-16T04:42:00.306824+0000","last_became_active":"2024-09-16T04:41:52.207190+0000","last_became_peered":"2024-09-16T04:41:52.207190+0000","last_unstale":"2024-09-16T04:42:00.306824+0000","last_undegraded":"2024-09-16T04:42:00.306824+0000","last_fullsized":"2024-09-16T04:42:00.306824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:41:59.065203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.065203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:24.124904+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0.001819048,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]}],"pool_stats":[{"poolid":3,"num_pg":25,"stat_sum":{"num_bytes":37076992,"num_objects":71,"num_object_clones":21,"num_object_copies":497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":3,"num_read":172,"num_read_kb":20509,"num_write":183,"num_write_kb":23472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64483328,"data_stored":85106688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":175,"acting":175,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36564,"kb_used_data":5372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335276,"statfs":{"total":96636764160,"available":96599322624,"internally_reserved":0,"allocated":5500928,"data_stored":6734992,"data_compressed":113769,"data_compressed_allocated":155648,"data_compressed_original":741376,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247820,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30668,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341172,"statfs":{"total":96636764160,"available":96605360128,"internally_reserved":0,"allocated":3727360,"data_stored":3977840,"data_compressed":81824,"data_compressed_allocated":114688,"data_compressed_original":229376,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247820,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":4756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":4870144,"data_stored":6156912,"data_compressed":81824,"data_compressed_allocated":114688,"data_compressed_original":229376,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313226,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":5444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":5574656,"data_stored":7233760,"data_compressed":69709,"data_compressed_allocated":98304,"data_compressed_original":196608,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313226,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30688,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341152,"statfs":{"total":96636764160,"available":96605339648,"internally_reserved":0,"allocated":3743744,"data_stored":4266338,"data_compressed":72746,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,0,0,0,0,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280523,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":5033984,"data_stored":6274118,"data_compressed":110750,"data_compressed_allocated":151552,"data_compressed_original":733184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280522,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31020,"kb_used_data":3984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340820,"statfs":{"total":96636764160,"available":96604999680,"internally_reserved":0,"allocated":4079616,"data_stored":4766246,"data_compressed":78805,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280523,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30984,"kb_used_data":3948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340856,"statfs":{"total":96636764160,"available":96605036544,"internally_reserved":0,"allocated":4042752,"data_stored":5376569,"data_compressed":78805,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280523,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37120,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334720,"statfs":{"total":96636764160,"available":96598753280,"internally_reserved":0,"allocated":6066176,"data_stored":7340460,"data_compressed":75763,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280523,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30532,"kb_used_data":3500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341308,"statfs":{"total":96636764160,"available":96605499392,"internally_reserved":0,"allocated":3584000,"data_stored":4243884,"data_compressed":75763,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280522,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31200,"kb_used_data":4164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340640,"statfs":{"total":96636764160,"available":96604815360,"internally_reserved":0,"allocated":4263936,"data_stored":4735404,"data_compressed":75763,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34964,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336876,"statfs":{"total":96636764160,"available":96600961024,"internally_reserved":0,"allocated":3858432,"data_stored":4633004,"data_compressed":75763,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":5328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":5455872,"data_stored":7606700,"data_compressed":75763,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280523,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31692,"kb_used_data":4656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340148,"statfs":{"total":96636764160,"available":96604311552,"internally_reserved":0,"allocated":4767744,"data_stored":6588966,"data_compressed":78805,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28900,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342940,"statfs":{"total":96636764160,"available":96607170560,"internally_reserved":0,"allocated":1904640,"data_stored":2275897,"data_compressed":78805,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280523,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35432,"kb_used_data":4236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336408,"statfs":{"total":96636764160,"available":96600481792,"internally_reserved":0,"allocated":4337664,"data_stored":5575084,"data_compressed":75763,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5361664,"data_stored":7237632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4640768,"data_stored":6193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4440064,"data_stored":5410816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3506176,"data_stored":5042176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5840896,"data_stored":7512064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3063808,"data_stored":4059136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3743744,"data_stored":4698112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3338240,"data_stored":4509696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5382144,"data_stored":7847936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3526656,"data_stored":3977216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4235264,"data_stored":6393856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3330048,"data_stored":4046848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1368064,"data_stored":2088960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4894720,"data_stored":5804032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4263936,"data_stored":5853184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3547136,"data_stored":4431872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:01.160 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T04:42:01.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-16T04:42:01.172 INFO:tasks.rados.rados.0.smithi165.stdout:147: done (8 left) 2024-09-16T04:42:01.172 INFO:tasks.rados.rados.0.smithi165.stdout:148: done (7 left) 2024-09-16T04:42:01.172 INFO:tasks.rados.rados.0.smithi165.stdout:151: done (6 left) 2024-09-16T04:42:01.172 INFO:tasks.rados.rados.0.smithi165.stdout:155: done (5 left) 2024-09-16T04:42:01.172 INFO:tasks.rados.rados.0.smithi165.stdout:156: done (4 left) 2024-09-16T04:42:01.172 INFO:tasks.rados.rados.0.smithi165.stdout:157: read oid 22 snap -1 2024-09-16T04:42:01.172 INFO:tasks.rados.rados.0.smithi165.stdout:157: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-16T04:42:01.173 INFO:tasks.rados.rados.0.smithi165.stdout:158: read oid 1 snap -1 2024-09-16T04:42:01.173 INFO:tasks.rados.rados.0.smithi165.stdout:158: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:01.173 INFO:tasks.rados.rados.0.smithi165.stdout:159: snap_create 2024-09-16T04:42:01.206 INFO:tasks.rados.rados.0.smithi165.stdout:154: finishing write tid 5 to smithi16530038-18 2024-09-16T04:42:01.206 INFO:tasks.rados.rados.0.smithi165.stdout:154: oid 18 updating version 0 to 62 2024-09-16T04:42:01.206 INFO:tasks.rados.rados.0.smithi165.stdout:154: oid 18 version 62 is already newer than 59 2024-09-16T04:42:01.206 INFO:tasks.rados.rados.0.smithi165.stdout:154: oid 18 version 62 is already newer than 60 2024-09-16T04:42:01.206 INFO:tasks.rados.rados.0.smithi165.stdout:154: oid 18 version 62 is already newer than 61 2024-09-16T04:42:01.206 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 62 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-09-16T04:42:01.206 INFO:tasks.rados.rados.0.smithi165.stdout:154: left oid 18 (ObjNum 76 snap 6 seq_num 76) 2024-09-16T04:42:01.207 INFO:tasks.rados.rados.0.smithi165.stdout:153: finishing write tid 4 to smithi16530038-16 2024-09-16T04:42:01.207 INFO:tasks.rados.rados.0.smithi165.stdout:153: oid 16 updating version 0 to 14 2024-09-16T04:42:01.207 INFO:tasks.rados.rados.0.smithi165.stdout:153: oid 16 updating version 14 to 15 2024-09-16T04:42:01.207 INFO:tasks.rados.rados.0.smithi165.stdout:153: oid 16 updating version 15 to 16 2024-09-16T04:42:01.207 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 16 (ObjNum 75 snap 6 seq_num 75) dirty exists 2024-09-16T04:42:01.207 INFO:tasks.rados.rados.0.smithi165.stdout:153: left oid 16 (ObjNum 75 snap 6 seq_num 75) 2024-09-16T04:42:01.225 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:01.219+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:01.232 INFO:tasks.rados.rados.0.smithi165.stdout:150: finishing copy_from to smithi16530038-41 2024-09-16T04:42:01.232 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 12 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T04:42:01.277 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:01.277 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:01.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:42:01.326 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:01.324+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:01.331 INFO:tasks.rados.rados.0.smithi165.stdout:149: finishing write tid 5 to smithi16530038-8 2024-09-16T04:42:01.331 INFO:tasks.rados.rados.0.smithi165.stdout:149: oid 8 updating version 0 to 18 2024-09-16T04:42:01.331 INFO:tasks.rados.rados.0.smithi165.stdout:149: oid 8 version 18 is already newer than 17 2024-09-16T04:42:01.331 INFO:tasks.rados.rados.0.smithi165.stdout:149: oid 8 updating version 18 to 21 2024-09-16T04:42:01.331 INFO:tasks.rados.rados.0.smithi165.stdout:149: oid 8 version 21 is already newer than 19 2024-09-16T04:42:01.332 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 21 (ObjNum 74 snap 5 seq_num 74) dirty exists 2024-09-16T04:42:01.332 INFO:tasks.rados.rados.0.smithi165.stdout:149: left oid 8 (ObjNum 74 snap 5 seq_num 74) 2024-09-16T04:42:01.426 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:01.423+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:01.450 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:01.450 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:01.464 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:01.481 INFO:tasks.rados.rados.0.smithi165.stdout:157: expect (ObjNum 66 snap 4 seq_num 66) 2024-09-16T04:42:01.527 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:01.523+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:01.580 INFO:tasks.rados.rados.0.smithi165.stdout:158: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:01.591 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:01.591 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:01.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:01.649 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:01.648+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:01.729 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:01.730 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:01.730 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:01.728+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:01.744 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:01.830 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:01.827+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:01.913 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:01.913 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:01.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:01.930 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:01.927+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:02.031 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:02.028+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:02.053 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:02.054 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:02.069 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:02.132 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:02.127+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:02.169 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pg_num to 26 2024-09-16T04:42:02.173 INFO:tasks.rados.rados.0.smithi165.stdout:149: done (6 left) 2024-09-16T04:42:02.173 INFO:tasks.rados.rados.0.smithi165.stdout:150: done (5 left) 2024-09-16T04:42:02.173 INFO:tasks.rados.rados.0.smithi165.stdout:153: done (4 left) 2024-09-16T04:42:02.173 INFO:tasks.rados.rados.0.smithi165.stdout:154: done (3 left) 2024-09-16T04:42:02.174 INFO:tasks.rados.rados.0.smithi165.stdout:157: done (2 left) 2024-09-16T04:42:02.174 INFO:tasks.rados.rados.0.smithi165.stdout:158: done (1 left) 2024-09-16T04:42:02.174 INFO:tasks.rados.rados.0.smithi165.stdout:159: done (0 left) 2024-09-16T04:42:02.174 INFO:tasks.rados.rados.0.smithi165.stdout:160: copy_from oid 48 from oid 42 current snap is 8 2024-09-16T04:42:02.174 INFO:tasks.rados.rados.0.smithi165.stdout:161: delete oid 29 current snap is 8 2024-09-16T04:42:02.174 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:42:02.202 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:02.202 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:02.217 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:02.233 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:02.227+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:02.283 INFO:tasks.rados.rados.0.smithi165.stdout:161: done (1 left) 2024-09-16T04:42:02.283 INFO:tasks.rados.rados.0.smithi165.stdout:162: snap_create 2024-09-16T04:42:02.333 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:02.332+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:02.389 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:02.389 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:02.400 INFO:tasks.rados.rados.0.smithi165.stdout:160: finishing copy_from racing read to smithi16530038-48 2024-09-16T04:42:02.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:02.453 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:02.452+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:02.536 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:02.532+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:02.570 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:02.570 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:02.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:42:02.602 INFO:tasks.rados.rados.0.smithi165.stdout:160: finishing copy_from to smithi16530038-48 2024-09-16T04:42:02.602 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 49 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:42:02.637 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:02.631+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:02.739 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:02.736+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:02.750 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:02.750 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:02.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:42:02.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:02.840 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:02.835+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:02.886 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:02.886 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:02.902 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:42:02.949 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:02.948+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:02.951 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:02.951 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:02.972 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:02.972 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:02.994 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:03.014 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:03.031 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:03.036 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:03.041 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:03.040+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:03.051 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:42:03.052 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:03.052 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:03.073 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:03.074 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:03.094 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:03.095 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:03.111 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:03.112 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:03.133 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:03.133 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:03.142 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:03.140+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:03.155 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:03.170 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:03.170 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:03.175 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:03.183 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:42:03.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:42:03.189 INFO:tasks.rados.rados.0.smithi165.stdout:160: done (1 left) 2024-09-16T04:42:03.189 INFO:tasks.rados.rados.0.smithi165.stdout:162: done (0 left) 2024-09-16T04:42:03.189 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 43 current snap is 9 2024-09-16T04:42:03.189 INFO:tasks.rados.rados.0.smithi165.stdout:163: seq_num 78 ranges {720896=557056,1277952=671744,1949696=98304} 2024-09-16T04:42:03.198 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:03.199 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:03.213 INFO:tasks.rados.rados.0.smithi165.stdout:163: writing smithi16530038-43 from 720896 to 1277952 tid 1 2024-09-16T04:42:03.222 INFO:tasks.rados.rados.0.smithi165.stdout:163: writing smithi16530038-43 from 1277952 to 1949696 tid 2 2024-09-16T04:42:03.225 INFO:tasks.rados.rados.0.smithi165.stdout:163: writing smithi16530038-43 from 1949696 to 2048000 tid 3 2024-09-16T04:42:03.225 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 10 current snap is 9 2024-09-16T04:42:03.225 INFO:tasks.rados.rados.0.smithi165.stdout:164: seq_num 79 ranges {163840=573440,737280=425984,1163264=344064} 2024-09-16T04:42:03.225 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:03.234 INFO:tasks.rados.rados.0.smithi165.stdout:164: writing smithi16530038-10 from 163840 to 737280 tid 1 2024-09-16T04:42:03.241 INFO:tasks.rados.rados.0.smithi165.stdout:164: writing smithi16530038-10 from 737280 to 1163264 tid 2 2024-09-16T04:42:03.243 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:03.240+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:03.245 INFO:tasks.rados.rados.0.smithi165.stdout:164: writing smithi16530038-10 from 1163264 to 1507328 tid 3 2024-09-16T04:42:03.245 INFO:tasks.rados.rados.0.smithi165.stdout:165: snap_create 2024-09-16T04:42:03.248 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:03.271 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:03.271 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:03.320 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:03.320 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:03.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:03.320 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:03.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.321 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_repop(client.4356.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-16T04:42:03.321 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:42:03.321 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.811880977999998, 2024-09-16T04:42:03.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021469340000000001, 2024-09-16T04:42:03.322 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.322 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.322 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:03.322 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:42:03.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:42:03.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505667+0000", 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505677+0000", 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3090000000000001e-06 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505703+0000", 2024-09-16T04:42:03.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6139000000000002e-05 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505720+0000", 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7329e-05 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505886+0000", 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016637499999999999 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.505945+0000", 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8699999999999997e-05 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.507713+0000", 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001768218 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:28.507814+0000", 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100274 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.327 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_repop(client.4356.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:29.620732+0000", 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.696815186000002, 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00184699, 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620732+0000", 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620732+0000", 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620734+0000", 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8199999999999999e-06 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620747+0000", 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2323e-05 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620775+0000", 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8824999999999999e-05 2024-09-16T04:42:03.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620791+0000", 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6056e-05 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.620970+0000", 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017873800000000001 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.621012+0000", 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1653999999999998e-05 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.622489+0000", 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014767059999999999 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:29.622579+0000", 2024-09-16T04:42:03.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0867999999999998e-05 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 37/32 ECSubWrite(tid=34, reqid=client.4728.0:169, at_version=37'11, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:47.573967+0000", 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.743580767999999, 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102186843, 2024-09-16T04:42:03.333 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.573967+0000", 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.573967+0000", 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.573969+0000", 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0540000000000002e-06 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.574032+0000", 2024-09-16T04:42:03.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2860999999999993e-05 2024-09-16T04:42:03.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.574074+0000", 2024-09-16T04:42:03.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1928000000000002e-05 2024-09-16T04:42:03.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.574098+0000", 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4189000000000002e-05 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.674354+0000", 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100256153 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.674369+0000", 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5342e-05 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.676085+0000", 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001715762 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.676154+0000", 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8553999999999994e-05 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 38/32 ECSubWrite(tid=36, reqid=client.4728.0:188, at_version=38'13, trim_to=0'0, roll_forward_to=37'11))", 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.179477+0000", 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.138070298000001, 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10118962500000001, 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.179477+0000", 2024-09-16T04:42:03.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.179477+0000", 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.179478+0000", 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.179619+0000", 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014104499999999999 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.179656+0000", 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7021999999999999e-05 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.179685+0000", 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8903000000000001e-05 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278119+0000", 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098433940999999997 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278144+0000", 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5418999999999998e-05 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280583+0000", 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024387480000000001 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280667+0000", 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3803999999999999e-05 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.345 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 38/32 ECSubWrite(tid=33, reqid=client.4728.0:181, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.272884+0000", 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.044663324, 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102436214, 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272884+0000", 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272884+0000", 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272886+0000", 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.273116+0000", 2024-09-16T04:42:03.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000229931 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.273147+0000", 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1699000000000003e-05 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.273172+0000", 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4389e-05 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.373657+0000", 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10048555200000001 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.373667+0000", 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8009999999999998e-06 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.375271+0000", 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001603527 2024-09-16T04:42:03.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.375320+0000", 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9795e-05 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 38/32 ECSubWrite(tid=34, reqid=client.4728.0:182, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.277177+0000", 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.040370639000001, 2024-09-16T04:42:03.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101968824, 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277177+0000", 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277177+0000", 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277179+0000", 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9970000000000001e-06 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277391+0000", 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021181499999999999 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277441+0000", 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0430999999999999e-05 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377525+0000", 2024-09-16T04:42:03.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10008368300000001 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377604+0000", 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9330000000000001e-05 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377613+0000", 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6659999999999997e-06 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.379126+0000", 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015134510000000001 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.379146+0000", 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9451000000000001e-05 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 38/32 ECSubWrite(tid=35, reqid=client.4728.0:183, at_version=38'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.377566+0000", 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.939981492999999, 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0017819979999999999, 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377566+0000", 2024-09-16T04:42:03.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377566+0000", 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377567+0000", 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8400000000000003e-07 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377597+0000", 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.968e-05 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377612+0000", 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5262999999999999e-05 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377620+0000", 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4239999999999993e-06 2024-09-16T04:42:03.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.378235+0000", 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00061508199999999998 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.378243+0000", 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3799999999999996e-06 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.379311+0000", 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010687610000000001 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.379348+0000", 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6624e-05 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=56, reqid=osd.11.0:56, at_version=44'19, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.119152+0000", 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.1983955949999991, 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037895310000000001, 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119152+0000", 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119152+0000", 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119152+0000", 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4000000000000002e-07 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119192+0000", 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9517999999999997e-05 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119225+0000", 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3473000000000003e-05 2024-09-16T04:42:03.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119244+0000", 2024-09-16T04:42:03.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8508999999999998e-05 2024-09-16T04:42:03.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119337+0000", 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2769000000000006e-05 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.119350+0000", 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.358e-05 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.122895+0000", 2024-09-16T04:42:03.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035447180000000001 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.122942+0000", 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6523999999999997e-05 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.bs3 44/32 ECSubRead(tid=58, to_read={3:d797f264:::smithi16530038-12:head=0,49152,0}, subchunks={3:d797f264:::smithi16530038-12:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.315433+0000", 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.002114465, 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10030818800000001, 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.315433+0000", 2024-09-16T04:42:03.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.315433+0000", 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.315434+0000", 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5300000000000004e-07 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.315439+0000", 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3839999999999996e-06 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.315449+0000", 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6709999999999994e-06 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.315458+0000", 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4020000000000008e-06 2024-09-16T04:42:03.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.415614+0000", 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10015526199999999 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.415741+0000", 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000127816 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.371 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=64, reqid=osd.11.0:59, at_version=0'0, trim_to=0'0, roll_forward_to=44'21))", 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.520506+0000", 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.7970414019999996, 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10161100200000001, 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520506+0000", 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520506+0000", 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520507+0000", 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8100000000000001e-07 2024-09-16T04:42:03.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520520+0000", 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2449999999999999e-05 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520535+0000", 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5264000000000002e-05 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.620622+0000", 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100087539 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.620782+0000", 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015924600000000001 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.620812+0000", 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0015999999999999e-05 2024-09-16T04:42:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.622052+0000", 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012404619999999999 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.622117+0000", 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5043999999999996e-05 2024-09-16T04:42:03.376 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.16s4,RELEASE,e:47,from:0(0),reservation_nonce:0)", 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:57.978332+0000", 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.3392154930000002, 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100317767, 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:03.377 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:57.978332+0000", 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:57.978332+0000", 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:57.978333+0000", 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4399999999999999e-07 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:57.978338+0000", 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2639999999999999e-06 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:57.978351+0000", 2024-09-16T04:42:03.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.309e-05 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:57.978363+0000", 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2211999999999999e-05 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.078531+0000", 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100167111 2024-09-16T04:42:03.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.078595+0000", 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4041999999999994e-05 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.078650+0000", 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5303999999999998e-05 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.381 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=29, reqid=client.4728.0:365, at_version=49'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.0461391369999999, 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15627164299999999, 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:03.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271409+0000", 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:42:03.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271765+0000", 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00035584100000000002 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271795+0000", 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9595999999999999e-05 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271820+0000", 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5301999999999999e-05 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425575+0000", 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15375470799999999 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425593+0000", 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427644+0000", 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020504630000000002 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427680+0000", 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6613999999999999e-05 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.386 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=81, reqid=client.4728.0:376, at_version=49'25, trim_to=0'0, roll_forward_to=45'23))", 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.362180+0000", 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9553677060000001, 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001717496, 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362180+0000", 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362180+0000", 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362181+0000", 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4300000000000002e-07 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362211+0000", 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.991e-05 2024-09-16T04:42:03.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362231+0000", 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0574e-05 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362243+0000", 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1531e-05 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362327+0000", 2024-09-16T04:42:03.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4497000000000001e-05 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.362338+0000", 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1055e-05 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.363856+0000", 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001517221 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.363897+0000", 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1865e-05 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=83, reqid=client.4728.0:376, at_version=0'0, trim_to=0'0, roll_forward_to=49'25))", 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.792771+0000", 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.5247761139999998, 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016654300000000001, 2024-09-16T04:42:03.391 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.792771+0000", 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.792771+0000", 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.792774+0000", 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.334e-06 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.792805+0000", 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1241999999999997e-05 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.792850+0000", 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5222000000000002e-05 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.792878+0000", 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7342999999999999e-05 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.793068+0000", 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019044799999999999 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.793102+0000", 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3989000000000003e-05 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.794324+0000", 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001221955 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.794437+0000", 2024-09-16T04:42:03.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000112897 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=31, reqid=client.4728.0:377, at_version=49'15, trim_to=0'0, roll_forward_to=49'14))", 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.917814+0000", 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.3997339740000001, 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019999420000000002, 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.396 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.917814+0000", 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.917814+0000", 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.917816+0000", 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3700000000000002e-06 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.917869+0000", 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3257999999999997e-05 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.917910+0000", 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0252e-05 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.917929+0000", 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9185999999999999e-05 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.918096+0000", 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016702500000000001 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.918116+0000", 2024-09-16T04:42:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.919722+0000", 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001605481 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.919814+0000", 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1756999999999996e-05 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.400 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=32, reqid=client.4728.0:377, at_version=0'0, trim_to=0'0, roll_forward_to=49'15))", 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:00.069208+0000", 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.2483398370000001, 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101124478, 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.401 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.069208+0000", 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.069208+0000", 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.069209+0000", 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2449999999999999e-06 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.069225+0000", 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5846999999999999e-05 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.069243+0000", 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7672999999999998e-05 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.169327+0000", 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100084399 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.169441+0000", 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000114096 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.169458+0000", 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7379000000000001e-05 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.170287+0000", 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00082891599999999996 2024-09-16T04:42:03.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.170332+0000", 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4922999999999997e-05 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 50/32 ECSubWrite(tid=34, reqid=client.4728.0:385, at_version=50'17, trim_to=0'0, roll_forward_to=49'15))", 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:00.204550+0000", 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.1129971150000002, 2024-09-16T04:42:03.405 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026530389999999998, 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.204550+0000", 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.204550+0000", 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.204552+0000", 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3960000000000001e-06 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.204923+0000", 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000371233 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.204951+0000", 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.834e-05 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.204967+0000", 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5914e-05 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.205070+0000", 2024-09-16T04:42:03.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010213399999999999 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.205082+0000", 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2945999999999999e-05 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.207162+0000", 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002079558 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.207204+0000", 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1517999999999998e-05 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 50/32 ECSubWrite(tid=35, reqid=client.4728.0:386, at_version=50'18, trim_to=0'0, roll_forward_to=49'15))", 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:00.209177+0000", 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.1083701640000001, 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101895481, 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.410 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209177+0000", 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209177+0000", 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209178+0000", 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3900000000000003e-07 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209453+0000", 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027417499999999998 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.412 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209471+0000", 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8575000000000001e-05 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.309558+0000", 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100086831 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.309617+0000", 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8621e-05 2024-09-16T04:42:03.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.309623+0000", 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7580000000000003e-06 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.311032+0000", 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001408833 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.311073+0000", 2024-09-16T04:42:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0748999999999999e-05 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 50/32 ECSubWrite(tid=36, reqid=client.4728.0:387, at_version=50'19, trim_to=0'0, roll_forward_to=49'15))", 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:00.312407+0000", 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.0051403890000001, 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001716593, 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.415 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312407+0000", 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312407+0000", 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.416 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312408+0000", 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3900000000000003e-07 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312472+0000", 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3588000000000002e-05 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312488+0000", 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6005999999999999e-05 2024-09-16T04:42:03.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312497+0000", 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2939999999999994e-06 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312584+0000", 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7342999999999994e-05 2024-09-16T04:42:03.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.312597+0000", 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2306999999999999e-05 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.314073+0000", 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014762410000000001 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.314124+0000", 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0875000000000003e-05 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 50/32 ECSubWrite(tid=88, reqid=client.4728.0:395, at_version=50'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:00.408915+0000", 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.908632178, 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030146349999999999, 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.420 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.408915+0000", 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.408915+0000", 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.421 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.408916+0000", 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.46e-07 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.409260+0000", 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000343899 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.409308+0000", 2024-09-16T04:42:03.422 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8018e-05 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.409326+0000", 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8498000000000001e-05 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.409561+0000", 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00023446799999999999 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.409571+0000", 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0003999999999999e-05 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.411881+0000", 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002310304 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.411930+0000", 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8998000000000001e-05 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.424 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.425 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:03.425 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:03.425 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:03.426 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:42:03.427 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:03.423+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:03.528 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:03.524+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:03.531 INFO:tasks.rados.rados.0.smithi165.stdout:163: finishing write tid 1 to smithi16530038-43 2024-09-16T04:42:03.531 INFO:tasks.rados.rados.0.smithi165.stdout:163: finishing write tid 2 to smithi16530038-43 2024-09-16T04:42:03.559 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:03.559 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:03.559 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:03.559 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:03.573 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:42:03.628 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:03.624+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:03.632 INFO:tasks.rados.rados.0.smithi165.stdout:163: finishing write tid 3 to smithi16530038-43 2024-09-16T04:42:03.698 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:03.698 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:03.698 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:03.712 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:42:03.732 INFO:tasks.rados.rados.0.smithi165.stdout:163: finishing write tid 4 to smithi16530038-43 2024-09-16T04:42:03.738 INFO:tasks.rados.rados.0.smithi165.stdout:164: finishing write tid 1 to smithi16530038-10 2024-09-16T04:42:03.738 INFO:tasks.rados.rados.0.smithi165.stdout:164: finishing write tid 2 to smithi16530038-10 2024-09-16T04:42:03.759 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:03.755+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:03.830 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:03.823+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:03.833 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:03.833 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:03.833 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:03.833 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:03.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.833 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=10, reqid=client.4728.0:122, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:03.834 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:03.834 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.145484537000002, 2024-09-16T04:42:03.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201524707, 2024-09-16T04:42:03.834 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.834 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.834 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.834 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:03.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684051+0000", 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.277e-06 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684081+0000", 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0414999999999999e-05 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684109+0000", 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7455000000000001e-05 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684125+0000", 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5707000000000001e-05 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884504+0000", 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200379487 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884519+0000", 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4953000000000001e-05 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885504+0000", 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098537600000000005 2024-09-16T04:42:03.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885574+0000", 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0036999999999994e-05 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.855162937999999, 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20260964000000001, 2024-09-16T04:42:03.839 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974373+0000", 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.683e-06 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974841+0000", 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468255 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974891+0000", 2024-09-16T04:42:03.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0198000000000003e-05 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.074965+0000", 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100073226 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175170+0000", 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100205119 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175184+0000", 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4626e-05 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176898+0000", 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017136390000000001 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176981+0000", 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2893999999999996e-05 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.843 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=16, reqid=client.4728.0:131, at_version=36'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.754519647999999, 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102541787, 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075016+0000", 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.06e-06 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075046+0000", 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0485999999999999e-05 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075066+0000", 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9572000000000002e-05 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075078+0000", 2024-09-16T04:42:03.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2626000000000001e-05 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176189+0000", 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10111062699999999 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176204+0000", 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4543e-05 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.177494+0000", 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012900450000000001 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.177556+0000", 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2828000000000002e-05 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=11, reqid=client.4728.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.087873+0000", 2024-09-16T04:42:03.848 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.741661318999999, 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101857478, 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087873+0000", 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087873+0000", 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087875+0000", 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.013e-06 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088207+0000", 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00033206000000000002 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088233+0000", 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5885999999999999e-05 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088248+0000", 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5126000000000001e-05 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088401+0000", 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015341399999999999 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088416+0000", 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.435e-05 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189673+0000", 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101257364 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189730+0000", 2024-09-16T04:42:03.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7265000000000001e-05 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=12, reqid=client.4728.0:134, at_version=36'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.088282+0000", 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.741252442, 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102194618, 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088282+0000", 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088282+0000", 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088283+0000", 2024-09-16T04:42:03.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8000000000000005e-07 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088303+0000", 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0801999999999998e-05 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088316+0000", 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:42:03.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.088325+0000", 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189373+0000", 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10104791 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.189388+0000", 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4772e-05 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.190406+0000", 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001018337 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.190476+0000", 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0177000000000003e-05 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:160, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.375871327999999, 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20191688999999999, 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453664+0000", 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453695+0000", 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0998999999999999e-05 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453714+0000", 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9745000000000002e-05 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.553888+0000", 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10017351100000001 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654137+0000", 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100248949 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654151+0000", 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4279e-05 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655526+0000", 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013750520000000001 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655580+0000", 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3634999999999997e-05 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s1 36/32 ECSubWrite(tid=21, reqid=client.4728.0:156, at_version=36'9, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:42:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.554011+0000", 2024-09-16T04:42:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.275522983999998, 2024-09-16T04:42:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102169199, 2024-09-16T04:42:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554011+0000", 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554011+0000", 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554012+0000", 2024-09-16T04:42:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0359999999999999e-06 2024-09-16T04:42:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554201+0000", 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000188963 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554225+0000", 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4202e-05 2024-09-16T04:42:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.554239+0000", 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3246e-05 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654468+0000", 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10022879 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654496+0000", 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8204999999999998e-05 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.656122+0000", 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016259930000000001 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.656180+0000", 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8764000000000002e-05 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s1 36/32 ECSubWrite(tid=22, reqid=client.4728.0:157, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.244168985999998, 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102232006, 2024-09-16T04:42:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:42:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585367+0000", 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585532+0000", 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164928 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585561+0000", 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9391e-05 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585578+0000", 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6569999999999999e-05 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.685858+0000", 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100280496 2024-09-16T04:42:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.685876+0000", 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7499999999999998e-05 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.687532+0000", 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016559509999999999 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.687597+0000", 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5456999999999993e-05 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 38/32 ECSubWrite(tid=20, reqid=client.4728.0:176, at_version=38'12, trim_to=0'0, roll_forward_to=36'9))", 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.493290+0000", 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.336243984999999, 2024-09-16T04:42:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10168057, 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.493290+0000", 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.493290+0000", 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.493291+0000", 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1999999999999998e-07 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.493598+0000", 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00030672300000000001 2024-09-16T04:42:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.493614+0000", 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5622999999999999e-05 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.493624+0000", 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9760000000000007e-06 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.593842+0000", 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.1002185 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.593851+0000", 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1940000000000004e-06 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.594889+0000", 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001038031 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.594971+0000", 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1602999999999998e-05 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s3 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.409215+0000", 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.420318868000001, 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10267963099999999, 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409215+0000", 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409215+0000", 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409217+0000", 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4869999999999999e-06 2024-09-16T04:42:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409262+0000", 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5488999999999998e-05 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409289+0000", 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6682e-05 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409305+0000", 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5976000000000001e-05 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.509553+0000", 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100247613 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.509580+0000", 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6872e-05 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.511814+0000", 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022344019999999999 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.511895+0000", 2024-09-16T04:42:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1110000000000001e-05 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.19s1 42/41 ECSubRead(tid=32, to_read={3:9e9173cd:::smithi16530038-4:head=0,16384,0}, subchunks={3:9e9173cd:::smithi16530038-4:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.414144+0000", 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.415389983000001, 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.1022832, 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414144+0000", 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414144+0000", 2024-09-16T04:42:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414145+0000", 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.023e-06 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414159+0000", 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3359e-05 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414186+0000", 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7118999999999999e-05 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516004+0000", 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101818013 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516187+0000", 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018360100000000001 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516427+0000", 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024008499999999999 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.19s1 42/41 ECSubWrite(tid=35, reqid=client.4728.0:262, at_version=42'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.517939+0000", 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.311595411000001, 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103147299, 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517939+0000", 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517939+0000", 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517940+0000", 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9699999999999995e-07 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517999+0000", 2024-09-16T04:42:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9308000000000002e-05 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518020+0000", 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1305999999999999e-05 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518040+0000", 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0123e-05 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518525+0000", 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00048423699999999999 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518546+0000", 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1265999999999999e-05 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.620987+0000", 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102440958 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621086+0000", 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9303999999999997e-05 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.19s1 42/41 ECSubWrite(tid=36, reqid=client.4728.0:263, at_version=42'14, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.518116+0000", 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.31141815, 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10640616899999999, 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518116+0000", 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518116+0000", 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518117+0000", 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7300000000000001e-07 2024-09-16T04:42:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518186+0000", 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8981999999999998e-05 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518208+0000", 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2705e-05 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518228+0000", 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0129999999999999e-05 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619775+0000", 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101546368 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619800+0000", 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4879999999999999e-05 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.624447+0000", 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046478400000000003 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.624522+0000", 2024-09-16T04:42:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4790999999999996e-05 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s3 42/41 ECSubWrite(tid=23, reqid=client.4728.0:197, at_version=42'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.310479681, 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102207279, 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519168+0000", 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113026 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519192+0000", 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3189000000000001e-05 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519208+0000", 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6725999999999999e-05 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619474+0000", 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100266022 2024-09-16T04:42:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619502+0000", 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7429000000000001e-05 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621190+0000", 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016884529999999999 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621262+0000", 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1643999999999994e-05 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.14s3 42/41 ECSubRead(tid=26, to_read={3:28a14731:::smithi16530038-3:head=0,12288,0}, subchunks={3:28a14731:::smithi16530038-3:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.629212+0000", 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.200322310000001, 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10175906899999999, 2024-09-16T04:42:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.629212+0000", 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.629212+0000", 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.629214+0000", 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.697e-06 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.629232+0000", 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8692999999999999e-05 2024-09-16T04:42:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.629275+0000", 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2215000000000002e-05 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.629309+0000", 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4102e-05 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.730788+0000", 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10147890800000001 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.730971+0000", 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000183454 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.912 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.es2 43/32 ECSubRead(tid=21, to_read={3:741a05b8:::smithi16530038-36:head=0,147456,0}, subchunks={3:741a05b8:::smithi16530038-36:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:53.203899+0000", 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.625635276000001, 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102872009, 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203899+0000", 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203899+0000", 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203901+0000", 2024-09-16T04:42:03.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.807e-06 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203917+0000", 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5953000000000001e-05 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203949+0000", 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2149e-05 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.304066+0000", 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10011745399999999 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.306455+0000", 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002388879 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.306771+0000", 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000315767 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.as2 43/32 ECSubRead(tid=37, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.621119605000001, 2024-09-16T04:42:03.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200544685, 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.34e-07 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208425+0000", 2024-09-16T04:42:03.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8859999999999999e-06 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208457+0000", 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1644999999999999e-05 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.308556+0000", 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100099178 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.408782+0000", 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100226105 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.408959+0000", 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017743699999999999 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.12s5 43/41 ECSubWrite(tid=174, reqid=client.4728.0:257, at_version=43'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:53.308959+0000", 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.520575705000001, 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101862252, 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.922 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.308959+0000", 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.308959+0000", 2024-09-16T04:42:03.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.308960+0000", 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.037e-06 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.309018+0000", 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8145e-05 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.309046+0000", 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8129000000000001e-05 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.309068+0000", 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2425e-05 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.409305+0000", 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100236984 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.409330+0000", 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.410741+0000", 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001410639 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.410821+0000", 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9903999999999999e-05 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.927 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 48/32 ECSubWrite(tid=44, reqid=client.4728.0:362, at_version=48'17, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.4728480279999996, 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205456467, 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356687+0000", 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356985+0000", 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029765100000000001 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357014+0000", 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9853e-05 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357034+0000", 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9592000000000001e-05 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557308+0000", 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20027399500000001 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557322+0000", 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4346e-05 2024-09-16T04:42:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562072+0000", 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047490960000000004 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562143+0000", 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1191000000000003e-05 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.2s5 53/37 ECSubWrite(tid=192, reqid=client.4728.0:420, at_version=53'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:03.322544+0000", 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.50698986499999998, 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102605686, 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322544+0000", 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322544+0000", 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322546+0000", 2024-09-16T04:42:03.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322915+0000", 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00036882700000000001 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322956+0000", 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0926000000000001e-05 2024-09-16T04:42:03.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.423060+0000", 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100104049 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.423202+0000", 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141903 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.423219+0000", 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6849000000000001e-05 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.425059+0000", 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0018400280000000001 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.425150+0000", 2024-09-16T04:42:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1124000000000004e-05 2024-09-16T04:42:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:03.938 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:03.940 INFO:tasks.rados.rados.0.smithi165.stdout:163: finishing write tid 5 to smithi16530038-43 2024-09-16T04:42:03.940 INFO:tasks.rados.rados.0.smithi165.stdout:163: oid 43 updating version 0 to 9 2024-09-16T04:42:03.940 INFO:tasks.rados.rados.0.smithi165.stdout:163: oid 43 version 9 is already newer than 6 2024-09-16T04:42:03.940 INFO:tasks.rados.rados.0.smithi165.stdout:163: oid 43 version 9 is already newer than 7 2024-09-16T04:42:03.940 INFO:tasks.rados.rados.0.smithi165.stdout:163: oid 43 version 9 is already newer than 8 2024-09-16T04:42:03.940 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 9 (ObjNum 78 snap 9 seq_num 78) dirty exists 2024-09-16T04:42:03.940 INFO:tasks.rados.rados.0.smithi165.stdout:163: left oid 43 (ObjNum 78 snap 9 seq_num 78) 2024-09-16T04:42:03.941 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:42:03.941 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:03.936+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:04.041 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:04.035+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:04.112 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:04.112 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:04.112 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:04.113 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:04.126 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:42:04.141 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:04.140+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:04.197 INFO:tasks.rados.rados.0.smithi165.stdout:163: done (2 left) 2024-09-16T04:42:04.197 INFO:tasks.rados.rados.0.smithi165.stdout:165: done (1 left) 2024-09-16T04:42:04.197 INFO:tasks.rados.rados.0.smithi165.stdout:166: snap_create 2024-09-16T04:42:04.242 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:04.240+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:04.245 INFO:tasks.rados.rados.0.smithi165.stdout:164: finishing write tid 3 to smithi16530038-10 2024-09-16T04:42:04.245 INFO:tasks.rados.rados.0.smithi165.stdout:164: finishing write tid 4 to smithi16530038-10 2024-09-16T04:42:04.255 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:04.255 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:04.269 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:42:04.343 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:04.339+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:04.393 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4356.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 36.991750906999997, 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31207237399999999, 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.394 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4356", 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.59:0/2607617674", 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:04.396 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398126+0000", 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0999999999999998e-06 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398137+0000", 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0253e-05 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398158+0000", 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1401999999999999e-05 2024-09-16T04:42:04.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398176+0000", 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8552000000000001e-05 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597014+0000", 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19883803 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.398 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597043+0000", 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8683999999999998e-05 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.604764+0000", 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0077206829999999999 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605002+0000", 2024-09-16T04:42:04.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023771400000000001 2024-09-16T04:42:04.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 15", 2024-09-16T04:42:04.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605234+0000", 2024-09-16T04:42:04.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023226099999999999 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.606579+0000", 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013450650000000001 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710048+0000", 2024-09-16T04:42:04.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103468869 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710127+0000", 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9491000000000002e-05 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710197+0000", 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9270000000000006e-05 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.402 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 36/32 ECSubWrite(tid=26, reqid=client.4728.0:116, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.615394550000001, 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101615175, 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774481+0000", 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1999999999999998e-07 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774502+0000", 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0398999999999999e-05 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774516+0000", 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3822000000000001e-05 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.774525+0000", 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8269999999999993e-06 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.874737+0000", 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10021129300000001 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.874752+0000", 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5415999999999999e-05 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.875998+0000", 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001246236 2024-09-16T04:42:04.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.876096+0000", 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7361999999999994e-05 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 36/32 ECSubWrite(tid=26, reqid=client.4728.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.966685+0000", 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.423190049999999, 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20155758700000001, 2024-09-16T04:42:04.407 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966685+0000", 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966685+0000", 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966687+0000", 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1500000000000002e-06 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966710+0000", 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3272e-05 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.966742+0000", 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1275999999999997e-05 2024-09-16T04:42:04.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.066872+0000", 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100129973 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.167193+0000", 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100321778 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.167220+0000", 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7002e-05 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.168171+0000", 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095042200000000003 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.168243+0000", 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1713999999999998e-05 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 36/32 ECSubWrite(tid=29, reqid=client.4728.0:137, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.178719+0000", 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.211156041999999, 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10198995499999999, 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.412 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178719+0000", 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178719+0000", 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178720+0000", 2024-09-16T04:42:04.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3570000000000001e-06 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178763+0000", 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2781000000000003e-05 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178788+0000", 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4559e-05 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.178807+0000", 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8836000000000001e-05 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.279430+0000", 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100623613 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.279445+0000", 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5133000000000001e-05 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.415 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.280624+0000", 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011788930000000001 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.280709+0000", 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4783000000000005e-05 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:190 3.8s0 3:16f2ba44:::smithi16530038-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.28976673, 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70466111799999998, 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 190 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100109+0000", 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7300000000000004e-07 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100116+0000", 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3099999999999997e-06 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100128+0000", 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2262000000000001e-05 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100137+0000", 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480895+0000", 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.38075793800000002 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480902+0000", 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0700000000000001e-06 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582260+0000", 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101357841 2024-09-16T04:42:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582314+0000", 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4175000000000002e-05 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804615+0000", 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.22230154099999999 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804681+0000", 2024-09-16T04:42:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5356000000000002e-05 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804769+0000", 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8782000000000005e-05 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.980501944, 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.311529638, 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409375+0000", 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8899999999999999e-06 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409439+0000", 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4428e-05 2024-09-16T04:42:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409480+0000", 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0454999999999998e-05 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409505+0000", 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4620000000000001e-05 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619957+0000", 2024-09-16T04:42:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21045218900000001 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619979+0000", 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1866e-05 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720830+0000", 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100851396 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720903+0000", 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2793999999999995e-05 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=31, reqid=client.4728.0:186, at_version=42'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.975217068999999, 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.306308988, 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414660+0000", 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:42:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414693+0000", 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414716+0000", 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3356000000000001e-05 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414734+0000", 2024-09-16T04:42:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7787000000000001e-05 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621156+0000", 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.206422364 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621180+0000", 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3589999999999999e-05 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720911+0000", 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099731400999999997 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720967+0000", 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5841000000000003e-05 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=33, reqid=client.4728.0:187, at_version=42'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.431 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.516721+0000", 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.873154301, 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.204424734, 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516721+0000", 2024-09-16T04:42:04.432 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516721+0000", 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516723+0000", 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1069999999999999e-06 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516769+0000", 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5945000000000001e-05 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516801+0000", 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2289000000000002e-05 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516825+0000", 2024-09-16T04:42:04.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3405999999999999e-05 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.719877+0000", 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20305267599999999 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.719903+0000", 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6100000000000001e-05 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.721087+0000", 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00118366 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.721146+0000", 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8551000000000002e-05 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 42/37 ECSubWriteReply(tid=9, last_complete=42'6, committed=1, applied=1))", 2024-09-16T04:42:04.436 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.611588+0000", 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.778286693, 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20040545900000001, 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.611588+0000", 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.611588+0000", 2024-09-16T04:42:04.437 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.611590+0000", 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.703e-06 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.611599+0000", 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0529999999999996e-06 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.438 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.611620+0000", 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1060000000000002e-05 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.711708+0000", 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10008787099999999 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.811960+0000", 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10025242600000001 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.811994+0000", 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3346000000000002e-05 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=39, reqid=client.4728.0:187, at_version=0'0, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.922721+0000", 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.467154032, 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10209523600000001, 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.440 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.922721+0000", 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.922721+0000", 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.922723+0000", 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4399999999999999e-06 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.922751+0000", 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7336000000000001e-05 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.922791+0000", 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9888000000000002e-05 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.442 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.922815+0000", 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4079000000000001e-05 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.023082+0000", 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10026692800000001 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.023118+0000", 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6069000000000003e-05 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.443 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.024758+0000", 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001640201 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.024816+0000", 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8294999999999997e-05 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 44/37 ECSubWriteReply(tid=11, last_complete=44'8, committed=1, applied=1))", 2024-09-16T04:42:04.444 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.220993+0000", 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.168882557, 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200354383, 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.220993+0000", 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.220993+0000", 2024-09-16T04:42:04.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.220994+0000", 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.73e-06 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.221004+0000", 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0193e-05 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.446 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.221030+0000", 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5452999999999998e-05 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.221049+0000", 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9338999999999999e-05 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.421309+0000", 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20025943900000001 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.421347+0000", 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8229000000000003e-05 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 44/37 ECSubWriteReply(tid=11, last_complete=44'8, committed=1, applied=1))", 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.222602+0000", 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.167273191, 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19980350899999999, 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.448 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.222602+0000", 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.222602+0000", 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.222604+0000", 2024-09-16T04:42:04.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.626e-06 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.222614+0000", 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.097e-05 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.222645+0000", 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0083e-05 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.322740+0000", 2024-09-16T04:42:04.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100095619 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.421413+0000", 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.098672762999999997 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.422405+0000", 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099244799999999994 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.8s0 47/37 ECSubRead(tid=15, to_read={3:16f2ba44:::smithi16530038-21:head=0,192512,0}, subchunks={3:16f2ba44:::smithi16530038-21:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.363565+0000", 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.0263103769999997, 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201025976, 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.363565+0000", 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.452 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.363565+0000", 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.363565+0000", 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.363565+0000", 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.363641+0000", 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6197000000000002e-05 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.363673+0000", 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2396e-05 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.564419+0000", 2024-09-16T04:42:04.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20074610100000001 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.564591+0000", 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000171282 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.0254962250000004, 2024-09-16T04:42:04.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30496093299999999, 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.456 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364381+0000", 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.663e-06 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364619+0000", 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023814999999999999 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364642+0000", 2024-09-16T04:42:04.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2875999999999999e-05 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364655+0000", 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3849000000000001e-05 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.667942+0000", 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30328671699999998 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.669340+0000", 2024-09-16T04:42:04.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013976780000000001 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.364737+0000", 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.0251380760000002, 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201319568, 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.459 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364737+0000", 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364737+0000", 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364738+0000", 2024-09-16T04:42:04.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.5999999999999991e-07 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364983+0000", 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024521299999999998 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.365001+0000", 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7708999999999999e-05 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.365010+0000", 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4739999999999994e-06 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.565951+0000", 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200940692 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.566057+0000", 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010552000000000001 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.462 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=14, attrs_read=0))", 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.367269+0000", 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.0226064209999999, 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19901128600000001, 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.463 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.367269+0000", 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.367269+0000", 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.367271+0000", 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.367654+0000", 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00038357199999999999 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.367684+0000", 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9975999999999999e-05 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.467846+0000", 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100162077 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.566230+0000", 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.098384055999999998 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.566280+0000", 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9685000000000003e-05 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s5 47/37 ECSubWrite(tid=189, reqid=osd.13.0:189, at_version=47'9, trim_to=0'0, roll_forward_to=46'7))", 2024-09-16T04:42:04.466 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.438051+0000", 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.9518242849999998, 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20431723199999999, 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438051+0000", 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438051+0000", 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438052+0000", 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3399999999999998e-07 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438101+0000", 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8952000000000002e-05 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438129+0000", 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8498999999999999e-05 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438153+0000", 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3558999999999999e-05 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.638528+0000", 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200375478 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.638544+0000", 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.596e-05 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.642277+0000", 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037327660000000002 2024-09-16T04:42:04.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.642368+0000", 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1184e-05 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=14, attrs_read=0))", 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.463932+0000", 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.9259428239999998, 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102263268, 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.471 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.463932+0000", 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.463932+0000", 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.463932+0000", 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.463932+0000", 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.463995+0000", 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3132000000000005e-05 2024-09-16T04:42:04.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.464016+0000", 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.086e-05 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.566132+0000", 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10211571799999999 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.566196+0000", 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3558000000000004e-05 2024-09-16T04:42:04.474 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.475 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.467876+0000", 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.9219987439999997, 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19998988300000001, 2024-09-16T04:42:04.476 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.467876+0000", 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.467876+0000", 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.467877+0000", 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3300000000000001e-07 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.468171+0000", 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029365199999999999 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.468193+0000", 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2646e-05 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.468204+0000", 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0219e-05 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.667735+0000", 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.199531493 2024-09-16T04:42:04.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.667866+0000", 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013103999999999999 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDScrubReserve(3.10s4,RELEASE,e:49,from:14(0),reservation_nonce:0)", 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:59.373854+0000", 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.0160212870000001, 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19073784599999999, 2024-09-16T04:42:04.480 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373854+0000", 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373854+0000", 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373855+0000", 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4270000000000001e-06 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373864+0000", 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0929999999999992e-06 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373902+0000", 2024-09-16T04:42:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.7752e-05 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373922+0000", 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0270000000000001e-05 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.564518+0000", 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19059605600000001 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.564554+0000", 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5982000000000002e-05 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.564592+0000", 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.7265999999999999e-05 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.484 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:04.485 INFO:tasks.rados.rados.0.smithi165.stdout:164: finishing write tid 5 to smithi16530038-10 2024-09-16T04:42:04.485 INFO:tasks.rados.rados.0.smithi165.stdout:164: oid 10 updating version 0 to 27 2024-09-16T04:42:04.485 INFO:tasks.rados.rados.0.smithi165.stdout:164: oid 10 updating version 27 to 28 2024-09-16T04:42:04.485 INFO:tasks.rados.rados.0.smithi165.stdout:164: oid 10 updating version 28 to 29 2024-09-16T04:42:04.486 INFO:tasks.rados.rados.0.smithi165.stdout:164: oid 10 updating version 29 to 30 2024-09-16T04:42:04.486 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 30 (ObjNum 79 snap 9 seq_num 79) dirty exists 2024-09-16T04:42:04.486 INFO:tasks.rados.rados.0.smithi165.stdout:164: left oid 10 (ObjNum 79 snap 9 seq_num 79) 2024-09-16T04:42:04.487 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:42:04.487 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:04.484+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:04.586 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:04.584+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:04.621 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:04.621 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:04.621 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:04.621 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:04.636 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:42:04.687 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:04.684+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:04.765 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:04.765 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:04.765 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:04.765 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:04.765 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:04.774 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:42:04.819 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:04.815+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:04.889 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:04.883+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=11, reqid=client.4728.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.811932355, 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20239900399999999, 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.903 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087079+0000", 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5730000000000002e-06 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087392+0000", 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000313164 2024-09-16T04:42:04.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087425+0000", 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2768999999999997e-05 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187523+0000", 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100097504 2024-09-16T04:42:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187687+0000", 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164537 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187711+0000", 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4209000000000001e-05 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289394+0000", 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10168234399999999 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289476+0000", 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1903999999999994e-05 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 36/32 ECSubWrite(tid=27, reqid=client.4728.0:136, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.162018+0000", 2024-09-16T04:42:04.909 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.736991408000002, 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10274949799999999, 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162018+0000", 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162018+0000", 2024-09-16T04:42:04.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162020+0000", 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.187e-06 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162297+0000", 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027736200000000002 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.162328+0000", 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1248999999999998e-05 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262471+0000", 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100142278 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262628+0000", 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000157584 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.262676+0000", 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7924999999999999e-05 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.264719+0000", 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020430090000000001 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.264767+0000", 2024-09-16T04:42:04.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7904000000000001e-05 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=12, reqid=client.4728.0:134, at_version=36'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.187573+0000", 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.711436343999999, 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102594399, 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187573+0000", 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187573+0000", 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187574+0000", 2024-09-16T04:42:04.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0300000000000001e-06 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187604+0000", 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0565999999999998e-05 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187624+0000", 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9899e-05 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187636+0000", 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2353e-05 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.288921+0000", 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101284807 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.288944+0000", 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2858999999999999e-05 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.290094+0000", 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011501479999999999 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.290167+0000", 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2737000000000005e-05 2024-09-16T04:42:04.918 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.as3 36/32 ECSubRead(tid=13, to_read={3:5619390f:::smithi16530038-41:head=0,4096,0}, subchunks={3:5619390f:::smithi16530038-41:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.609435621999999, 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20090377300000001, 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.919 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289575+0000", 2024-09-16T04:42:04.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.717e-06 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289598+0000", 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3326e-05 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289642+0000", 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3174999999999999e-05 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.389821+0000", 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100179457 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490099+0000", 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10027802299999999 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490477+0000", 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037807500000000001 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/32 ECSubWrite(tid=33, reqid=client.4728.0:181, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.401186+0000", 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.497823365999999, 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103483412, 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401186+0000", 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401186+0000", 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401186+0000", 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.06e-07 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401499+0000", 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00031315500000000002 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401511+0000", 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.204e-05 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401527+0000", 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401651+0000", 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012476100000000001 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401667+0000", 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5673000000000001e-05 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.504607+0000", 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102939817 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.504669+0000", 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2014000000000006e-05 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.927 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/32 ECSubWrite(tid=34, reqid=client.4728.0:182, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.401550+0000", 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.497459014, 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103609434, 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401550+0000", 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401550+0000", 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401550+0000", 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1699999999999999e-07 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401649+0000", 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8863999999999994e-05 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401657+0000", 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7710000000000006e-06 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401663+0000", 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5520000000000004e-06 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.503602+0000", 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101939217 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.503619+0000", 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7238999999999999e-05 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.505132+0000", 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015128299999999999 2024-09-16T04:42:04.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.505159+0000", 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7543999999999999e-05 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/32 ECSubWrite(tid=35, reqid=client.4728.0:183, at_version=38'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.401675+0000", 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.497333701999999, 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10407501600000001, 2024-09-16T04:42:04.932 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401675+0000", 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401675+0000", 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401676+0000", 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.22e-07 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401685+0000", 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7939999999999997e-06 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401691+0000", 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5620000000000003e-06 2024-09-16T04:42:04.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.401695+0000", 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2549999999999999e-06 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.504739+0000", 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103043706 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.504753+0000", 2024-09-16T04:42:04.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3723000000000001e-05 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.505733+0000", 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098050500000000001 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.505750+0000", 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7149000000000001e-05 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s2 38/37 ECSubRead(tid=38, to_read={3:8583b9f2:::smithi16530038-39:head=0,159744,0}, subchunks={3:8583b9f2:::smithi16530038-39:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.405740+0000", 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.493269435999999, 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.198583595, 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.405740+0000", 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.405740+0000", 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.938 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.405740+0000", 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8400000000000004e-07 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.405749+0000", 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8899999999999996e-06 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.405762+0000", 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3178e-05 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.939 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.405771+0000", 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0909999999999999e-06 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.603625+0000", 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19785403500000001 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.604323+0000", 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00069771700000000004 2024-09-16T04:42:04.940 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 38/32 ECSubWrite(tid=18, reqid=client.4728.0:194, at_version=38'11, trim_to=0'0, roll_forward_to=36'9))", 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.411660151, 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202828813, 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487637+0000", 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00028784999999999998 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487673+0000", 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5542e-05 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.587811+0000", 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100137777 2024-09-16T04:42:04.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.688010+0000", 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10019921599999999 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.688025+0000", 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4955e-05 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.690131+0000", 2024-09-16T04:42:04.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002106442 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.690178+0000", 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6381000000000002e-05 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 38/37 ECSubWrite(tid=44, reqid=client.4728.0:212, at_version=38'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.887605+0000", 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.011404123999998, 2024-09-16T04:42:04.945 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102655345, 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887605+0000", 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887605+0000", 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.946 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887607+0000", 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.733e-06 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887663+0000", 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6137999999999997e-05 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887694+0000", 2024-09-16T04:42:04.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1801999999999998e-05 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887718+0000", 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3575999999999999e-05 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887878+0000", 2024-09-16T04:42:04.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016010699999999999 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.887907+0000", 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8789000000000002e-05 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.990207+0000", 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10229999200000001 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.990260+0000", 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3208000000000003e-05 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.17s3 42/41 ECSubRead(tid=53, to_read={3:ed82f61a:::smithi16530038-1:head=0,262144,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.614356+0000", 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.284653319, 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103026385, 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.614356+0000", 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.614356+0000", 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.951 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.614358+0000", 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.57e-06 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.614381+0000", 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2755000000000002e-05 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.614423+0000", 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1609000000000001e-05 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.614450+0000", 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7432000000000001e-05 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.715440+0000", 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100989765 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.715445+0000", 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4299999999999997e-06 2024-09-16T04:42:04.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.716358+0000", 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00091285900000000002 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.717382+0000", 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010239649999999999 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 44/41 ECSubWrite(tid=55, reqid=osd.4.0:55, at_version=44'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.685697033, 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.296361333, 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.955 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2e-07 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213370+0000", 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7278000000000003e-05 2024-09-16T04:42:04.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213391+0000", 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1696999999999999e-05 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213413+0000", 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1104999999999999e-05 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.407449+0000", 2024-09-16T04:42:04.957 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19403645 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.407462+0000", 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3315e-05 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509615+0000", 2024-09-16T04:42:04.958 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102152944 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509673+0000", 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8023999999999999e-05 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 44/41 ECSubWrite(tid=59, reqid=client.4728.0:299, at_version=44'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.675917566000001, 2024-09-16T04:42:04.959 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.28930476599999999, 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223094+0000", 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6359999999999998e-06 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223754+0000", 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00066019600000000005 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223785+0000", 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1165999999999999e-05 2024-09-16T04:42:04.961 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223806+0000", 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0134999999999999e-05 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.508219+0000", 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.28441328900000001 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.508234+0000", 2024-09-16T04:42:04.962 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5213e-05 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.512334+0000", 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041000660000000003 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.512396+0000", 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2064999999999996e-05 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.963 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/37 ECSubWrite(tid=60, reqid=client.4728.0:295, at_version=44'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.332932+0000", 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.566076857000001, 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.17678576900000001, 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.332932+0000", 2024-09-16T04:42:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.332932+0000", 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.332934+0000", 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.013e-06 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.965 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.333074+0000", 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014024600000000001 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.333105+0000", 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0698999999999999e-05 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.333122+0000", 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6552999999999999e-05 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.408042+0000", 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.074920630000000002 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.408059+0000", 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6405999999999998e-05 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509684+0000", 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10162502299999999 2024-09-16T04:42:04.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509718+0000", 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4199000000000002e-05 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 44/41 ECSubWrite(tid=61, reqid=client.4728.0:300, at_version=44'19, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.414538+0000", 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.484470594999999, 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102558423, 2024-09-16T04:42:04.968 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.414538+0000", 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.414538+0000", 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.414541+0000", 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9790000000000002e-06 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.414734+0000", 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000192807 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.414774+0000", 2024-09-16T04:42:04.970 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9755999999999997e-05 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.514868+0000", 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100093922 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.515039+0000", 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017096400000000001 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.971 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.515069+0000", 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9822e-05 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.516988+0000", 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019191130000000001 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.517097+0000", 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010906 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 49/32 ECSubWrite(tid=29, reqid=client.4728.0:365, at_version=49'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:59.270647+0000", 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.6283614829999999, 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103155118, 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.270647+0000", 2024-09-16T04:42:04.973 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.270647+0000", 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.270648+0000", 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.974 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.270900+0000", 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000252103 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.270933+0000", 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2239000000000001e-05 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.270958+0000", 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5502000000000001e-05 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.975 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.371397+0000", 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100438693 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.371422+0000", 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5775999999999999e-05 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373717+0000", 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022943939999999999 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:59.373803+0000", 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5714e-05 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/37 ECSubWrite(tid=84, reqid=client.4728.0:380, at_version=49'15, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:00.204478+0000", 2024-09-16T04:42:04.977 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.6945312980000002, 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.105153749, 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204478+0000", 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.978 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204478+0000", 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204479+0000", 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204953+0000", 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00047453399999999998 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204973+0000", 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.933e-05 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204986+0000", 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3706000000000001e-05 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.205140+0000", 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015364900000000001 2024-09-16T04:42:04.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.981 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.981 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.205165+0000", 2024-09-16T04:42:04.981 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5173000000000001e-05 2024-09-16T04:42:04.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.981 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.309576+0000", 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.104410979 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.309631+0000", 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5387999999999999e-05 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 50/32 ECSubWrite(tid=34, reqid=client.4728.0:385, at_version=50'17, trim_to=0'0, roll_forward_to=49'15))", 2024-09-16T04:42:04.982 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:00.204566+0000", 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.6944430199999996, 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103061206, 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204566+0000", 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204566+0000", 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204568+0000", 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.68e-06 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204940+0000", 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000372273 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.984 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204970+0000", 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9643000000000001e-05 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.204998+0000", 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8031999999999999e-05 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.205136+0000", 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013888299999999999 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.205151+0000", 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4876e-05 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.307566+0000", 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10241507499999999 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.307627+0000", 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0744e-05 2024-09-16T04:42:04.986 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/37 ECSubWrite(tid=85, reqid=client.4728.0:381, at_version=49'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:00.206326+0000", 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.6926826850000003, 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103355722, 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.987 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.206326+0000", 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.206326+0000", 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.206327+0000", 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:42:04.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.206544+0000", 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021747100000000001 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.206571+0000", 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6723000000000001e-05 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.206594+0000", 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2342000000000001e-05 2024-09-16T04:42:04.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.207318+0000", 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00072484000000000003 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.207343+0000", 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4709e-05 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.309639+0000", 2024-09-16T04:42:04.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102296268 2024-09-16T04:42:04.991 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.991 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.991 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.991 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.309682+0000", 2024-09-16T04:42:04.991 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2626000000000002e-05 2024-09-16T04:42:04.991 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/37 ECSubWrite(tid=86, reqid=client.4728.0:382, at_version=49'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:00.208035+0000", 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.6909743959999997, 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102577474, 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.992 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.208035+0000", 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.208035+0000", 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.208036+0000", 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7969999999999999e-06 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.208150+0000", 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113297 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.208178+0000", 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7942000000000001e-05 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.208204+0000", 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6423000000000001e-05 2024-09-16T04:42:04.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.308828+0000", 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10062446999999999 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.308849+0000", 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0679999999999999e-05 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.310527+0000", 2024-09-16T04:42:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016780370000000001 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:00.310612+0000", 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4827999999999995e-05 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:04.996 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:04.998 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:42:04.998 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:04.995+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:05.098 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:05.095+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:05.166 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:05.166 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:05.167 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:05.167 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:05.181 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:42:05.199 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:05.196+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:05.206 INFO:tasks.rados.rados.0.smithi165.stdout:164: done (1 left) 2024-09-16T04:42:05.206 INFO:tasks.rados.rados.0.smithi165.stdout:166: done (0 left) 2024-09-16T04:42:05.206 INFO:tasks.rados.rados.0.smithi165.stdout:167: read oid 42 snap -1 2024-09-16T04:42:05.206 INFO:tasks.rados.rados.0.smithi165.stdout:167: expect (ObjNum 54 snap 1 seq_num 54) 2024-09-16T04:42:05.206 INFO:tasks.rados.rados.0.smithi165.stdout:168: rmattr oid 17 current snap is 11 2024-09-16T04:42:05.206 INFO:tasks.rados.rados.0.smithi165.stdout:168: done (1 left) 2024-09-16T04:42:05.206 INFO:tasks.rados.rados.0.smithi165.stdout:169: setattr oid 28 current snap is 11 2024-09-16T04:42:05.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:42:05.211 INFO:tasks.rados.rados.0.smithi165.stdout:170: copy_from oid 33 from oid 16 current snap is 11 2024-09-16T04:42:05.211 INFO:tasks.rados.rados.0.smithi165.stdout:171: setattr oid 32 current snap is 11 2024-09-16T04:42:05.215 INFO:tasks.rados.rados.0.smithi165.stdout:172: snap_remove snap 6 2024-09-16T04:42:05.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:05.300 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:05.295+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:05.303 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:05.304 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:05.304 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:05.304 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:05.304 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:05.319 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:42:05.401 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:05.396+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:05.445 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 41.066252810999998, 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1755266710000001, 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.446 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374332+0000", 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8059999999999999e-06 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374345+0000", 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2529e-05 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374365+0000", 2024-09-16T04:42:05.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0695000000000001e-05 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374383+0000", 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7569000000000001e-05 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374514+0000", 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013070099999999999 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:05.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374544+0000", 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0731000000000001e-05 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.343896+0000", 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.969351224 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344169+0000", 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027390100000000001 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344583+0000", 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041308600000000001 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.346047+0000", 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014642699999999999 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549695+0000", 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203648463 2024-09-16T04:42:05.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549791+0000", 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6137999999999993e-05 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549856+0000", 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.452 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.819836631999998, 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51346886800000002, 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 111 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620747+0000", 2024-09-16T04:42:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620757+0000", 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0316000000000001e-05 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620778+0000", 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620792+0000", 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4362000000000001e-05 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928510+0000", 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.307717717 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928629+0000", 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011919 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928822+0000", 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019246199999999999 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930635+0000", 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00181292 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134065+0000", 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20343018399999999 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134149+0000", 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4635000000000005e-05 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134214+0000", 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4973000000000003e-05 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.458 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.819766268999999, 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51366009000000001, 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 112 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.459 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620817+0000", 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:42:05.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621035+0000", 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021731100000000001 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621061+0000", 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6744999999999999e-05 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621082+0000", 2024-09-16T04:42:05.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1061999999999999e-05 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929460+0000", 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.308377346 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929595+0000", 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013512600000000001 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929798+0000", 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020277199999999999 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932465+0000", 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002667862 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134319+0000", 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20185381599999999 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.463 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134407+0000", 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7894999999999998e-05 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134476+0000", 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8829000000000001e-05 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.464 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.819466325999997, 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51357863199999998, 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 113 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.465 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621118+0000", 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.689e-06 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621129+0000", 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621164+0000", 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5367999999999997e-05 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621182+0000", 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8352e-05 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930689+0000", 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.309507015 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930800+0000", 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000110871 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931026+0000", 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000226293 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.933569+0000", 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002542786 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134566+0000", 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200996272 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134640+0000", 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4412000000000005e-05 2024-09-16T04:42:05.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134694+0000", 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4524000000000002e-05 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.819369578999996, 2024-09-16T04:42:05.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51372108500000002, 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 114 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621214+0000", 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621225+0000", 2024-09-16T04:42:05.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1348999999999999e-05 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621245+0000", 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9966000000000002e-05 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621259+0000", 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4102e-05 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931642+0000", 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31038274300000002 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931776+0000", 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133552 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931973+0000", 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197262 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.033893+0000", 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101919751 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134774+0000", 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10088169600000001 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134847+0000", 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2309000000000002e-05 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.475 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134934+0000", 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6924999999999999e-05 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.819297769999999, 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51387291300000004, 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.476 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 115 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621286+0000", 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.26e-06 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621492+0000", 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020643500000000001 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621517+0000", 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621534+0000", 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6515999999999999e-05 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932515+0000", 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31098146799999998 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932631+0000", 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011636 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932814+0000", 2024-09-16T04:42:05.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000182406 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134236+0000", 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201422618 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135029+0000", 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079253199999999998 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135102+0000", 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3286000000000003e-05 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135157+0000", 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5043000000000001e-05 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:05.482 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.819015952000001, 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.717172162, 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 116 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:05.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621568+0000", 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621606+0000", 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8167999999999998e-05 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621628+0000", 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1742e-05 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621643+0000", 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5696e-05 2024-09-16T04:42:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235313+0000", 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61366968300000002 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235449+0000", 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136036 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235657+0000", 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207405 2024-09-16T04:42:05.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237049+0000", 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001392552 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338582+0000", 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101532626 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338670+0000", 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7888999999999998e-05 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338738+0000", 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8769000000000005e-05 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.488 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.818907961999997, 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.81671015999999996, 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 117 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.489 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621676+0000", 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4440000000000001e-06 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621686+0000", 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0845e-05 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621706+0000", 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9168999999999999e-05 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621723+0000", 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7538999999999999e-05 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236241+0000", 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61451815600000004 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236372+0000", 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013062300000000001 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236574+0000", 2024-09-16T04:42:05.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020165600000000001 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237570+0000", 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099642999999999997 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438224+0000", 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20065369 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438317+0000", 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3624999999999999e-05 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438384+0000", 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6983000000000006e-05 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.494 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.496 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.145833984999999, 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.508075677, 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 138 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.497 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294750+0000", 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294760+0000", 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294779+0000", 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8941999999999999e-05 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.499 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294794+0000", 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5038999999999999e-05 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405306+0000", 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11051159300000001 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405426+0000", 2024-09-16T04:42:05.500 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011977 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405611+0000", 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000185217 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709561+0000", 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394976699999998 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.501 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802549+0000", 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092988137999999998 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802707+0000", 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015861799999999999 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802824+0000", 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116517 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.502 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.145763861999999, 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60756662299999997, 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.503 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 139 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294820+0000", 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4330000000000001e-06 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294830+0000", 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0023e-05 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294850+0000", 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0622e-05 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294864+0000", 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4042e-05 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406110+0000", 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11124526 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406222+0000", 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111871 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406398+0000", 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017598000000000001 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709577+0000", 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30317971500000002 2024-09-16T04:42:05.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902101+0000", 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19252423499999999 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902267+0000", 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016537499999999999 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.508 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902385+0000", 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011806700000000001 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.145671734, 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60781196900000001, 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.509 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 140 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.510 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294912+0000", 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.407e-06 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295040+0000", 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012836600000000001 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295062+0000", 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1682e-05 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295083+0000", 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1109000000000001e-05 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507008+0000", 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21192486699999999 2024-09-16T04:42:05.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507149+0000", 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141042 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507369+0000", 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022046500000000001 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.513 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710298+0000", 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202928308 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902525+0000", 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.192226958 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902611+0000", 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6274999999999996e-05 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902722+0000", 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011149 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.145444153, 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70811914799999998, 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.515 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 141 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.516 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295140+0000", 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295152+0000", 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2053e-05 2024-09-16T04:42:05.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295172+0000", 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0282e-05 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295191+0000", 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8759000000000001e-05 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.518 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508486+0000", 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.213295545 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508630+0000", 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014419899999999999 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508824+0000", 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019358600000000001 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.519 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710318+0000", 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20149381 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.002981+0000", 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.29266302500000002 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003144+0000", 2024-09-16T04:42:05.520 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016330500000000001 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003257+0000", 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113064 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.480987424999999, 2024-09-16T04:42:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51415646199999998, 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 159 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959597+0000", 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9190000000000002e-06 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959609+0000", 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959629+0000", 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0035e-05 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959645+0000", 2024-09-16T04:42:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5786000000000001e-05 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167529+0000", 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.207883768 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167762+0000", 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023295499999999999 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.168134+0000", 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037181099999999999 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.270830+0000", 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102696683 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473585+0000", 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20275470700000001 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473677+0000", 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1772000000000002e-05 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473751+0000", 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4228999999999997e-05 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.480906662999999, 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51420228400000001, 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 160 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959677+0000", 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:42:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959687+0000", 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959711+0000", 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3811999999999998e-05 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959727+0000", 2024-09-16T04:42:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169193+0000", 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209466241 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169340+0000", 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000147279 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169545+0000", 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000205167 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374117+0000", 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20457175999999999 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473763+0000", 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099646306000000004 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473828+0000", 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4046000000000003e-05 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473878+0000", 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0241000000000002e-05 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.480829384, 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51424688600000001, 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 161 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959754+0000", 2024-09-16T04:42:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959920+0000", 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166088 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959944+0000", 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.536 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959963+0000", 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9176000000000001e-05 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371049+0000", 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41108644500000002 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371185+0000", 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013619899999999999 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371395+0000", 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020912 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.377708+0000", 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006313173 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473891+0000", 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.096182904 2024-09-16T04:42:05.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473949+0000", 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8016999999999997e-05 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474000+0000", 2024-09-16T04:42:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.480513627000001, 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51552797100000003, 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 162 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960070+0000", 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960077+0000", 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6830000000000008e-06 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960091+0000", 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3653e-05 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960101+0000", 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7629999999999995e-06 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372554+0000", 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41245356599999999 2024-09-16T04:42:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372785+0000", 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023023100000000001 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.373156+0000", 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000371601 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474010+0000", 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100854291 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475328+0000", 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001317304 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475477+0000", 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014954899999999999 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475596+0000", 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011913699999999999 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.480465399, 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51788103100000005, 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 163 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960118+0000", 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960124+0000", 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960134+0000", 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0115999999999999e-05 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960144+0000", 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374237+0000", 2024-09-16T04:42:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41409377600000002 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374462+0000", 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022478600000000001 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374797+0000", 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000334586 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474018+0000", 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099220947000000004 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477813+0000", 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003795171 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477921+0000", 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010822100000000001 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477998+0000", 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6662999999999994e-05 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.480426184999999, 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51822496799999995, 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 164 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960157+0000", 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7399999999999991e-07 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960241+0000", 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3677999999999994e-05 2024-09-16T04:42:05.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960256+0000", 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5206e-05 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960267+0000", 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1372999999999999e-05 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375650+0000", 2024-09-16T04:42:05.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41538323100000002 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375885+0000", 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000234758 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.376293+0000", 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040777599999999999 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474024+0000", 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097731440000000003 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478113+0000", 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004088225 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478261+0000", 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014825 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.557 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478381+0000", 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012005700000000001 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.480300111999998, 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51862660500000002, 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.558 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 165 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.559 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960283+0000", 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960289+0000", 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.49e-06 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960301+0000", 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2585999999999999e-05 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960309+0000", 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472635+0000", 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51232596699999999 2024-09-16T04:42:05.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472771+0000", 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136243 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473033+0000", 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026155699999999999 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:05.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474539+0000", 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015065 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478656+0000", 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041164319999999997 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478793+0000", 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013757299999999999 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478909+0000", 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011530400000000001 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:153 3.3s0 3:c31e3ec6:::smithi16530038-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 19.151000979999999, 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.511692229, 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:05.564 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 153 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289583+0000", 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.04e-06 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289590+0000", 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4899999999999997e-06 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289610+0000", 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9814999999999999e-05 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289625+0000", 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5553000000000002e-05 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.296844+0000", 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072185369999999997 2024-09-16T04:42:05.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297045+0000", 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020125199999999999 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599740+0000", 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.302695037 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599866+0000", 2024-09-16T04:42:05.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012644500000000001 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.801273+0000", 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20140706 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:05.569 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:05.570 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:05.571 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:05.571 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:05.572 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:05.572 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:42:05.573 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:05.573 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:05.574 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:05.574 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:05.574 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:05.574 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:05.575 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:05.568+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:05.575 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:05.575 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:05.593 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:05.593 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:05.610 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:05.610 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:05.665 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:05.665 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:05.672 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:05.667+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:05.691 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:05.707 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:05.707 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:05.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:05.720 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:05.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:42:05.738 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:05.738 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:05.759 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:05.759 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:05.773 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:05.771+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:05.777 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:05.777 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:05.800 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:05.801 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:05.841 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:05.841 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:05.854 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:42:05.874 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:05.871+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:05.975 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:05.972+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:05.978 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:05.978 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:05.992 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:42:06.075 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:06.072+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:06.159 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:06.159 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:06.175 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:42:06.176 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:06.171+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:06.209 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:42:06.214 INFO:tasks.rados.rados.0.smithi165.stdout:172: done (4 left) 2024-09-16T04:42:06.214 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 25 current snap is 11 2024-09-16T04:42:06.214 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 32 (ObjNum 55 snap 1 seq_num 55) dirty exists 2024-09-16T04:42:06.214 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 18 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-09-16T04:42:06.215 INFO:tasks.rados.rados.0.smithi165.stdout:167: expect (ObjNum 54 snap 1 seq_num 54) 2024-09-16T04:42:06.277 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:06.276+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:06.305 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:06.305 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:06.319 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:42:06.324 INFO:tasks.rados.rados.0.smithi165.stdout:170: finishing copy_from racing read to smithi16530038-33 2024-09-16T04:42:06.324 INFO:tasks.rados.rados.0.smithi165.stdout:173: seq_num 80 ranges {409600=589824,999424=573440,1572864=196608} 2024-09-16T04:42:06.340 INFO:tasks.rados.rados.0.smithi165.stdout:173: writing smithi16530038-25 from 409600 to 999424 tid 1 2024-09-16T04:42:06.350 INFO:tasks.rados.rados.0.smithi165.stdout:173: writing smithi16530038-25 from 999424 to 1572864 tid 2 2024-09-16T04:42:06.353 INFO:tasks.rados.rados.0.smithi165.stdout:173: writing smithi16530038-25 from 1572864 to 1769472 tid 3 2024-09-16T04:42:06.353 INFO:tasks.rados.rados.0.smithi165.stdout:167: done (4 left) 2024-09-16T04:42:06.353 INFO:tasks.rados.rados.0.smithi165.stdout:169: done (3 left) 2024-09-16T04:42:06.353 INFO:tasks.rados.rados.0.smithi165.stdout:171: done (2 left) 2024-09-16T04:42:06.353 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 29 current snap is 11 2024-09-16T04:42:06.354 INFO:tasks.rados.rados.0.smithi165.stdout:174: seq_num 81 ranges {0=606208,606208=720896,1327104=49152} 2024-09-16T04:42:06.364 INFO:tasks.rados.rados.0.smithi165.stdout:174: writing smithi16530038-29 from 0 to 606208 tid 1 2024-09-16T04:42:06.377 INFO:tasks.rados.rados.0.smithi165.stdout:174: writing smithi16530038-29 from 606208 to 1327104 tid 2 2024-09-16T04:42:06.378 INFO:tasks.rados.rados.0.smithi165.stdout:174: writing smithi16530038-29 from 1327104 to 1376256 tid 3 2024-09-16T04:42:06.378 INFO:tasks.rados.rados.0.smithi165.stdout:175: snap_remove snap 1 2024-09-16T04:42:06.379 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:06.371+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:06.445 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:06.445 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:06.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:42:06.479 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:06.476+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:06.580 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:06.576+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:06.589 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:06.589 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:06.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:42:06.680 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:06.675+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:06.773 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:06.773 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:06.782 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:06.779+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:06.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:42:06.884 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:06.879+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:06.910 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:06.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:42:06.984 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:06.979+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:07.049 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:07.063 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:42:07.110 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:07.108+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:07.185 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:07.179+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:07.186 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:07.197 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:42:07.197 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:42:07.197 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T04:42:07.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:07.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:42:07.230 INFO:tasks.rados.rados.0.smithi165.stdout:175: done (3 left) 2024-09-16T04:42:07.230 INFO:tasks.rados.rados.0.smithi165.stdout:176: delete oid 30 current snap is 11 2024-09-16T04:42:07.230 INFO:tasks.rados.rados.0.smithi165.stdout:170: finishing copy_from to smithi16530038-33 2024-09-16T04:42:07.230 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 16 (ObjNum 75 snap 6 seq_num 75) dirty exists 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:173: finishing write tid 1 to smithi16530038-25 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: finishing write tid 1 to smithi16530038-29 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: finishing write tid 2 to smithi16530038-29 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: finishing write tid 3 to smithi16530038-29 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: finishing write tid 4 to smithi16530038-29 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:173: finishing write tid 2 to smithi16530038-25 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:173: finishing write tid 3 to smithi16530038-25 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:173: finishing write tid 4 to smithi16530038-25 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: finishing write tid 5 to smithi16530038-29 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: oid 29 updating version 0 to 33 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: oid 29 updating version 33 to 35 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: oid 29 version 35 is already newer than 32 2024-09-16T04:42:07.231 INFO:tasks.rados.rados.0.smithi165.stdout:174: oid 29 version 35 is already newer than 34 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 35 (ObjNum 81 snap 11 seq_num 81) dirty exists 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:174: left oid 29 (ObjNum 81 snap 11 seq_num 81) 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:173: finishing write tid 5 to smithi16530038-25 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:173: oid 25 updating version 0 to 71 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:173: oid 25 version 71 is already newer than 68 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:173: oid 25 version 71 is already newer than 69 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:173: oid 25 version 71 is already newer than 70 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 71 (ObjNum 80 snap 11 seq_num 80) dirty exists 2024-09-16T04:42:07.232 INFO:tasks.rados.rados.0.smithi165.stdout:173: left oid 25 (ObjNum 80 snap 11 seq_num 80) 2024-09-16T04:42:07.286 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:07.283+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:07.366 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:07.367 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:07.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:42:07.386 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:07.384+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:07.487 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:07.484+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:07.516 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:07.517 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:07.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:42:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:07.533 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:42:07.555 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-16T04:42:07.220218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46359091,"num_objects":88,"num_object_clones":33,"num_object_copies":596,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":86,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":4,"num_read":308,"num_read_kb":20294,"num_write":442,"num_write_kb":37726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":186,"acting":186,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":543808,"kb_used_data":86360,"kb_used_omap":24,"kb_used_meta":457063,"kb_avail":1509405632,"statfs":{"total":1546188226560,"available":1545631367168,"internally_reserved":0,"allocated":88432640,"data_stored":107655917,"data_compressed":1538614,"data_compressed_allocated":2142208,"data_compressed_original":5144576,"omap_allocated":25522,"internal_metadata":468032590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8650752,"num_objects":9,"num_object_clones":8,"num_object_copies":63,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":86,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":-2,"num_read":5,"num_read_kb":-91,"num_write":50,"num_write_kb":10832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2801664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.035532"},"pg_stats":[{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:42:05.192801+0000","last_change":"2024-09-16T04:42:05.192801+0000","last_active":"2024-09-16T04:42:05.192801+0000","last_peered":"2024-09-16T04:42:05.192801+0000","last_clean":"2024-09-16T04:42:05.192801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:42:05.192801+0000","last_undegraded":"2024-09-16T04:42:05.192801+0000","last_fullsized":"2024-09-16T04:42:05.192801+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:05.192801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:42:05.192801+0000","last_clean_scrub_stamp":"2024-09-16T04:42:05.192801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:42:05.192801+0000","last_change":"2024-09-16T04:42:05.192801+0000","last_active":"2024-09-16T04:42:05.192801+0000","last_peered":"2024-09-16T04:42:05.192801+0000","last_clean":"2024-09-16T04:42:05.192801+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:42:05.192801+0000","last_undegraded":"2024-09-16T04:42:05.192801+0000","last_fullsized":"2024-09-16T04:42:05.192801+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:05.192801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:42:05.192801+0000","last_clean_scrub_stamp":"2024-09-16T04:42:05.192801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"44'6","reported_seq":13,"reported_epoch":54,"state":"active+degraded","last_fresh":"2024-09-16T04:42:04.604573+0000","last_change":"2024-09-16T04:42:04.604172+0000","last_active":"2024-09-16T04:42:04.604573+0000","last_peered":"2024-09-16T04:42:04.604573+0000","last_clean":"2024-09-16T04:41:55.235410+0000","last_became_active":"2024-09-16T04:42:04.604172+0000","last_became_peered":"2024-09-16T04:42:04.604172+0000","last_unstale":"2024-09-16T04:42:04.604573+0000","last_undegraded":"2024-09-16T04:42:04.196336+0000","last_fullsized":"2024-09-16T04:42:04.604573+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'6","last_scrub_stamp":"2024-09-16T04:42:00.511037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.511037+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.001369632,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":["0(3)","4(0)","5(5)","6(1)","8(4)","9(2)","11(6)"],"object_location_counts":[{"shards":"0(3),4(0),5(5),6(1),8(4),9(2),11(6)","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.16","version":"35'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:42:02.181824+0000","last_change":"2024-09-16T04:41:59.161483+0000","last_active":"2024-09-16T04:42:02.181824+0000","last_peered":"2024-09-16T04:42:02.181824+0000","last_clean":"2024-09-16T04:42:02.181824+0000","last_became_active":"2024-09-16T04:41:52.305704+0000","last_became_peered":"2024-09-16T04:41:52.305704+0000","last_unstale":"2024-09-16T04:42:02.181824+0000","last_undegraded":"2024-09-16T04:42:02.181824+0000","last_fullsized":"2024-09-16T04:42:02.181824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:57.977558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:57.977558+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:21.660624+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.0019716429999999999,"stat_sum":{"num_bytes":81920,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.15","version":"36'6","reported_seq":46,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:42:05.527280+0000","last_change":"2024-09-16T04:42:05.528055+0000","last_active":"2024-09-16T04:42:05.527280+0000","last_peered":"2024-09-16T04:42:05.527280+0000","last_clean":"2024-09-16T04:42:05.527280+0000","last_became_active":"2024-09-16T04:42:05.419352+0000","last_became_peered":"2024-09-16T04:42:05.419352+0000","last_unstale":"2024-09-16T04:42:05.527280+0000","last_undegraded":"2024-09-16T04:42:05.527280+0000","last_fullsized":"2024-09-16T04:42:05.527280+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053553209999999997,"stat_sum":{"num_bytes":761856,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.14","version":"49'28","reported_seq":14,"reported_epoch":54,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T04:42:04.731415+0000","last_change":"2024-09-16T04:42:04.731415+0000","last_active":"2024-09-16T04:42:04.731415+0000","last_peered":"2024-09-16T04:42:04.731415+0000","last_clean":"2024-09-16T04:41:59.690724+0000","last_became_active":"2024-09-16T04:42:04.626975+0000","last_became_peered":"2024-09-16T04:42:04.626975+0000","last_unstale":"2024-09-16T04:42:04.731415+0000","last_undegraded":"2024-09-16T04:42:04.205198+0000","last_fullsized":"2024-09-16T04:42:04.731415+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":899,"objects_trimmed":0,"snaptrim_duration":0.0055386970000000004,"stat_sum":{"num_bytes":3555328,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":["0(4)","1(3)","2(1)","4(2)","8(0)","13(6)","14(5)"],"object_location_counts":[{"shards":"0(4),1(3),2(1),4(2),8(0),13(6),14(5)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.295929+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:42:04.295929+0000","last_peered":"2024-09-16T04:42:04.295929+0000","last_clean":"2024-09-16T04:42:04.295929+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:42:04.295929+0000","last_undegraded":"2024-09-16T04:42:04.295929+0000","last_fullsized":"2024-09-16T04:42:04.295929+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:42:02.181947+0000","last_change":"2024-09-16T04:41:59.260646+0000","last_active":"2024-09-16T04:42:02.181947+0000","last_peered":"2024-09-16T04:42:02.181947+0000","last_clean":"2024-09-16T04:42:02.181947+0000","last_became_active":"2024-09-16T04:41:48.483004+0000","last_became_peered":"2024-09-16T04:41:48.483004+0000","last_unstale":"2024-09-16T04:42:02.181947+0000","last_undegraded":"2024-09-16T04:42:02.181947+0000","last_fullsized":"2024-09-16T04:42:02.181947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:59.307610+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.00079436899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.201913+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:42:04.201913+0000","last_peered":"2024-09-16T04:42:04.201913+0000","last_clean":"2024-09-16T04:42:04.201913+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:42:04.201913+0000","last_undegraded":"2024-09-16T04:42:04.201913+0000","last_fullsized":"2024-09-16T04:42:04.201913+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":112,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.306941+0000","last_change":"2024-09-16T04:42:00.916951+0000","last_active":"2024-09-16T04:42:04.306941+0000","last_peered":"2024-09-16T04:42:04.306941+0000","last_clean":"2024-09-16T04:42:04.306941+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:42:04.306941+0000","last_undegraded":"2024-09-16T04:42:04.306941+0000","last_fullsized":"2024-09-16T04:42:04.306941+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:42:00.916431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.916431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:12.657893+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.003216081,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.208057+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:42:04.208057+0000","last_peered":"2024-09-16T04:42:04.208057+0000","last_clean":"2024-09-16T04:42:04.208057+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:42:04.208057+0000","last_undegraded":"2024-09-16T04:42:04.208057+0000","last_fullsized":"2024-09-16T04:42:04.208057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"36'10","reported_seq":131,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.202798+0000","last_change":"2024-09-16T04:41:59.999328+0000","last_active":"2024-09-16T04:42:04.202798+0000","last_peered":"2024-09-16T04:42:04.202798+0000","last_clean":"2024-09-16T04:42:04.202798+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:42:04.202798+0000","last_undegraded":"2024-09-16T04:42:04.202798+0000","last_fullsized":"2024-09-16T04:42:04.202798+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:59.997918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.997918+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:12.200144+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.82029922,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":70,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.209381+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:42:04.209381+0000","last_peered":"2024-09-16T04:42:04.209381+0000","last_clean":"2024-09-16T04:42:04.209381+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:42:04.209381+0000","last_undegraded":"2024-09-16T04:42:04.209381+0000","last_fullsized":"2024-09-16T04:42:04.209381+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"50'17","reported_seq":138,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.297407+0000","last_change":"2024-09-16T04:41:59.269998+0000","last_active":"2024-09-16T04:42:04.297407+0000","last_peered":"2024-09-16T04:42:04.297407+0000","last_clean":"2024-09-16T04:42:04.297407+0000","last_became_active":"2024-09-16T04:41:48.475648+0000","last_became_peered":"2024-09-16T04:41:48.475648+0000","last_unstale":"2024-09-16T04:42:04.297407+0000","last_undegraded":"2024-09-16T04:42:04.297407+0000","last_fullsized":"2024-09-16T04:42:04.297407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:19.969104+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.001245527,"stat_sum":{"num_bytes":4046848,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":674,"num_write":1,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.3","version":"47'9","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:42:05.204835+0000","last_change":"2024-09-16T04:41:59.172465+0000","last_active":"2024-09-16T04:42:05.204835+0000","last_peered":"2024-09-16T04:42:05.204835+0000","last_clean":"2024-09-16T04:42:05.204835+0000","last_became_active":"2024-09-16T04:41:48.281236+0000","last_became_peered":"2024-09-16T04:41:48.281236+0000","last_unstale":"2024-09-16T04:42:05.204835+0000","last_undegraded":"2024-09-16T04:42:05.204835+0000","last_fullsized":"2024-09-16T04:42:05.204835+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:41:49.293927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.293927+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:05.086697+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.0024804720000000001,"stat_sum":{"num_bytes":1933312,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":1072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.2","version":"28'2","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.205842+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:42:04.205842+0000","last_peered":"2024-09-16T04:42:04.205842+0000","last_clean":"2024-09-16T04:42:04.205842+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:42:04.205842+0000","last_undegraded":"2024-09-16T04:42:04.205842+0000","last_fullsized":"2024-09-16T04:42:04.205842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":114,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.296626+0000","last_change":"2024-09-16T04:41:59.275435+0000","last_active":"2024-09-16T04:42:04.296626+0000","last_peered":"2024-09-16T04:42:04.296626+0000","last_clean":"2024-09-16T04:42:04.296626+0000","last_became_active":"2024-09-16T04:41:48.383143+0000","last_became_peered":"2024-09-16T04:41:48.383143+0000","last_unstale":"2024-09-16T04:42:04.296626+0000","last_undegraded":"2024-09-16T04:42:04.296626+0000","last_fullsized":"2024-09-16T04:42:04.296626+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:49.599894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.599894+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:13.631998+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0.00234228,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:42:05.304856+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:42:05.304856+0000","last_peered":"2024-09-16T04:42:05.304856+0000","last_clean":"2024-09-16T04:42:05.304856+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:42:05.304856+0000","last_undegraded":"2024-09-16T04:42:05.304856+0000","last_fullsized":"2024-09-16T04:42:05.304856+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"50'18","reported_seq":139,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.209053+0000","last_change":"2024-09-16T04:41:59.289706+0000","last_active":"2024-09-16T04:42:04.209053+0000","last_peered":"2024-09-16T04:42:04.209053+0000","last_clean":"2024-09-16T04:42:04.209053+0000","last_became_active":"2024-09-16T04:41:48.404175+0000","last_became_peered":"2024-09-16T04:41:48.404175+0000","last_unstale":"2024-09-16T04:42:04.209053+0000","last_undegraded":"2024-09-16T04:42:04.209053+0000","last_fullsized":"2024-09-16T04:42:04.209053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:50.183380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.183380+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:08.903519+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0.0041603789999999996,"stat_sum":{"num_bytes":2605056,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":1874,"num_write":6,"num_write_kb":1424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.205220+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:42:04.205220+0000","last_peered":"2024-09-16T04:42:04.205220+0000","last_clean":"2024-09-16T04:42:04.205220+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:42:04.205220+0000","last_undegraded":"2024-09-16T04:42:04.205220+0000","last_fullsized":"2024-09-16T04:42:04.205220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"53'9","reported_seq":123,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:42:05.305883+0000","last_change":"2024-09-16T04:41:59.171135+0000","last_active":"2024-09-16T04:42:05.305883+0000","last_peered":"2024-09-16T04:42:05.305883+0000","last_clean":"2024-09-16T04:42:05.305883+0000","last_became_active":"2024-09-16T04:41:48.477867+0000","last_became_peered":"2024-09-16T04:41:48.477867+0000","last_unstale":"2024-09-16T04:42:05.305883+0000","last_undegraded":"2024-09-16T04:42:05.305883+0000","last_fullsized":"2024-09-16T04:42:05.305883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.564481+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:02.463330+0000","scrub_duration":1083,"objects_trimmed":0,"snaptrim_duration":0.0016201869999999999,"stat_sum":{"num_bytes":2048000,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"2.3","version":"26'1","reported_seq":72,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.295229+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:42:04.295229+0000","last_peered":"2024-09-16T04:42:04.295229+0000","last_clean":"2024-09-16T04:42:04.295229+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:42:04.295229+0000","last_undegraded":"2024-09-16T04:42:04.295229+0000","last_fullsized":"2024-09-16T04:42:04.295229+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":267,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T04:42:05.204506+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:42:05.204506+0000","last_peered":"2024-09-16T04:42:05.204506+0000","last_clean":"2024-09-16T04:42:05.204506+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:42:05.204506+0000","last_undegraded":"2024-09-16T04:42:05.204506+0000","last_fullsized":"2024-09-16T04:42:05.204506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"44'8","reported_seq":137,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.194645+0000","last_change":"2024-09-16T04:42:02.590948+0000","last_active":"2024-09-16T04:42:04.194645+0000","last_peered":"2024-09-16T04:42:04.194645+0000","last_clean":"2024-09-16T04:42:04.194645+0000","last_became_active":"2024-09-16T04:42:02.486406+0000","last_became_peered":"2024-09-16T04:42:02.486406+0000","last_unstale":"2024-09-16T04:42:04.194645+0000","last_undegraded":"2024-09-16T04:42:04.194645+0000","last_fullsized":"2024-09-16T04:42:04.194645+0000","mapping_epoch":32,"log_start":"44'8","ondisk_log_start":"44'8","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'4","last_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.115829+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:18.561479+0000","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0.0014869340000000001,"stat_sum":{"num_bytes":1015808,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2257,"num_write":1,"num_write_kb":752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.9","version":"50'30","reported_seq":155,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.196417+0000","last_change":"2024-09-16T04:42:00.006109+0000","last_active":"2024-09-16T04:42:04.196417+0000","last_peered":"2024-09-16T04:42:04.196417+0000","last_clean":"2024-09-16T04:42:04.196417+0000","last_became_active":"2024-09-16T04:41:58.352078+0000","last_became_peered":"2024-09-16T04:41:58.352078+0000","last_unstale":"2024-09-16T04:42:04.196417+0000","last_undegraded":"2024-09-16T04:42:04.196417+0000","last_fullsized":"2024-09-16T04:42:04.196417+0000","mapping_epoch":32,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:14.406870+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.73358709600000005,"stat_sum":{"num_bytes":3899392,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.a","version":"50'19","reported_seq":131,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.306003+0000","last_change":"2024-09-16T04:41:59.277142+0000","last_active":"2024-09-16T04:42:04.306003+0000","last_peered":"2024-09-16T04:42:04.306003+0000","last_clean":"2024-09-16T04:42:04.306003+0000","last_became_active":"2024-09-16T04:41:42.059056+0000","last_became_peered":"2024-09-16T04:41:42.059056+0000","last_unstale":"2024-09-16T04:42:04.306003+0000","last_undegraded":"2024-09-16T04:42:04.306003+0000","last_fullsized":"2024-09-16T04:42:04.306003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:45.294565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001071825,"stat_sum":{"num_bytes":1294336,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":3273,"num_write":33,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.b","version":"50'16","reported_seq":118,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.309199+0000","last_change":"2024-09-16T04:41:59.287790+0000","last_active":"2024-09-16T04:42:04.309199+0000","last_peered":"2024-09-16T04:42:04.309199+0000","last_clean":"2024-09-16T04:42:04.309199+0000","last_became_active":"2024-09-16T04:41:42.059548+0000","last_became_peered":"2024-09-16T04:41:42.059548+0000","last_unstale":"2024-09-16T04:42:04.309199+0000","last_undegraded":"2024-09-16T04:42:04.309199+0000","last_fullsized":"2024-09-16T04:42:04.309199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:57.111357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021014800000000002,"stat_sum":{"num_bytes":3080192,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":1928,"num_write":12,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.c","version":"49'25","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:42:02.177361+0000","last_change":"2024-09-16T04:41:59.179492+0000","last_active":"2024-09-16T04:42:02.177361+0000","last_peered":"2024-09-16T04:42:02.177361+0000","last_clean":"2024-09-16T04:42:02.177361+0000","last_became_active":"2024-09-16T04:41:42.057674+0000","last_became_peered":"2024-09-16T04:41:42.057674+0000","last_unstale":"2024-09-16T04:42:02.177361+0000","last_undegraded":"2024-09-16T04:42:02.177361+0000","last_fullsized":"2024-09-16T04:42:02.177361+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:07.545736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010568019999999999,"stat_sum":{"num_bytes":2359296,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":16,"num_read_kb":1543,"num_write":55,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.d","version":"52'64","reported_seq":169,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.212230+0000","last_change":"2024-09-16T04:41:59.878505+0000","last_active":"2024-09-16T04:42:04.212230+0000","last_peered":"2024-09-16T04:42:04.212230+0000","last_clean":"2024-09-16T04:42:04.212230+0000","last_became_active":"2024-09-16T04:41:42.053696+0000","last_became_peered":"2024-09-16T04:41:42.053696+0000","last_unstale":"2024-09-16T04:42:04.212230+0000","last_undegraded":"2024-09-16T04:42:04.212230+0000","last_fullsized":"2024-09-16T04:42:04.212230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:46.090166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.59346497300000001,"stat_sum":{"num_bytes":8437760,"num_objects":13,"num_object_clones":6,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":29,"num_read_kb":2796,"num_write":40,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.e","version":"50'21","reported_seq":125,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T04:42:04.210931+0000","last_change":"2024-09-16T04:41:59.161839+0000","last_active":"2024-09-16T04:42:04.210931+0000","last_peered":"2024-09-16T04:42:04.210931+0000","last_clean":"2024-09-16T04:42:04.210931+0000","last_became_active":"2024-09-16T04:41:42.158174+0000","last_became_peered":"2024-09-16T04:41:42.158174+0000","last_unstale":"2024-09-16T04:42:04.210931+0000","last_undegraded":"2024-09-16T04:42:04.210931+0000","last_fullsized":"2024-09-16T04:42:04.210931+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:51.836835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023586499999999999,"stat_sum":{"num_bytes":3063808,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":22,"num_read_kb":4088,"num_write":36,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.f","version":"52'29","reported_seq":121,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T04:42:02.177843+0000","last_change":"2024-09-16T04:41:59.863556+0000","last_active":"2024-09-16T04:42:02.177843+0000","last_peered":"2024-09-16T04:42:02.177843+0000","last_clean":"2024-09-16T04:42:02.177843+0000","last_became_active":"2024-09-16T04:41:41.950892+0000","last_became_peered":"2024-09-16T04:41:41.950892+0000","last_unstale":"2024-09-16T04:42:02.177843+0000","last_undegraded":"2024-09-16T04:42:02.177843+0000","last_fullsized":"2024-09-16T04:42:02.177843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:05.400571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.68469351499999997,"stat_sum":{"num_bytes":786432,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":20,"num_read_kb":1734,"num_write":14,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.10","version":"44'10","reported_seq":18,"reported_epoch":55,"state":"active+recovering","last_fresh":"2024-09-16T04:42:05.819112+0000","last_change":"2024-09-16T04:42:05.948791+0000","last_active":"2024-09-16T04:42:05.819112+0000","last_peered":"2024-09-16T04:42:05.819112+0000","last_clean":"2024-09-16T04:41:54.625087+0000","last_became_active":"2024-09-16T04:42:04.808967+0000","last_became_peered":"2024-09-16T04:42:04.808967+0000","last_unstale":"2024-09-16T04:42:05.819112+0000","last_undegraded":"2024-09-16T04:42:04.298307+0000","last_fullsized":"2024-09-16T04:42:05.819112+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:41:59.373567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.373567+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.002144857,"stat_sum":{"num_bytes":655360,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,0,11],"acting":[13,6,7,4,8,0,11],"avail_no_missing":["1(3)","2(1)","4(6)","5(4)","6(5)","14(0)","15(2)"],"object_location_counts":[{"shards":"0(5),4(3),6(1),7(2),8(4),11(6),13(0)","objects":-1},{"shards":"1(3),2(1),4(6),5(4),6(5),14(0),15(2)","objects":2}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.11","version":"47'24","reported_seq":14,"reported_epoch":54,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T04:42:04.627055+0000","last_change":"2024-09-16T04:42:04.627055+0000","last_active":"2024-09-16T04:42:04.627055+0000","last_peered":"2024-09-16T04:42:04.627055+0000","last_clean":"2024-09-16T04:41:57.566943+0000","last_became_active":"2024-09-16T04:42:04.623974+0000","last_became_peered":"2024-09-16T04:42:04.623974+0000","last_unstale":"2024-09-16T04:42:04.627055+0000","last_undegraded":"2024-09-16T04:42:04.204996+0000","last_fullsized":"2024-09-16T04:42:04.627055+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'24","last_scrub_stamp":"2024-09-16T04:41:58.297098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.297098+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004138325,"stat_sum":{"num_bytes":3465216,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":["2(5)","4(0)","5(3)","9(2)","10(4)","11(1)","13(6)"],"object_location_counts":[{"shards":"2(5),4(0),5(3),9(2),10(4),11(1),13(6)","objects":5}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.12","version":"44'10","reported_seq":26,"reported_epoch":55,"state":"active+recovering","last_fresh":"2024-09-16T04:42:05.538998+0000","last_change":"2024-09-16T04:42:05.223854+0000","last_active":"2024-09-16T04:42:05.538998+0000","last_peered":"2024-09-16T04:42:05.538998+0000","last_clean":"2024-09-16T04:41:54.412167+0000","last_became_active":"2024-09-16T04:42:04.406048+0000","last_became_peered":"2024-09-16T04:42:04.406048+0000","last_unstale":"2024-09-16T04:42:05.538998+0000","last_undegraded":"2024-09-16T04:42:05.538998+0000","last_fullsized":"2024-09-16T04:42:05.538998+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:41:58.453618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.453618+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001675791,"stat_sum":{"num_bytes":1482752,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2801664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":["9(0)","1(5)","2(1)","3(2)","6(3)","11(6)","13(0)","15(4)"],"object_location_counts":[{"shards":"1(5),2(1),3(2),6(3),9(0),11(6),13(0),15(4)","objects":1},{"shards":"3(5),4(4),6(3),7(1),8(2),9(0),15(6)","objects":1}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]},{"pgid":"3.13","version":"36'3","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T04:42:00.306824+0000","last_change":"2024-09-16T04:41:59.171359+0000","last_active":"2024-09-16T04:42:00.306824+0000","last_peered":"2024-09-16T04:42:00.306824+0000","last_clean":"2024-09-16T04:42:00.306824+0000","last_became_active":"2024-09-16T04:41:52.207190+0000","last_became_peered":"2024-09-16T04:41:52.207190+0000","last_unstale":"2024-09-16T04:42:00.306824+0000","last_undegraded":"2024-09-16T04:42:00.306824+0000","last_fullsized":"2024-09-16T04:42:00.306824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:41:59.065203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.065203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:24.124904+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0.001819048,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":45768704,"num_objects":84,"num_object_clones":33,"num_object_copies":588,"num_objects_missing_on_primary":17,"num_objects_missing":17,"num_objects_degraded":86,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":4,"num_read":172,"num_read_kb":20176,"num_write":237,"num_write_kb":34264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":79589376,"data_stored":101736448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":342,"ondisk_log_size":342,"up":168,"acting":168,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36924,"kb_used_data":5736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334916,"statfs":{"total":96636764160,"available":96598953984,"internally_reserved":0,"allocated":5873664,"data_stored":7220709,"data_compressed":128787,"data_compressed_allocated":176128,"data_compressed_original":782336,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":4532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":4640768,"data_stored":4881349,"data_compressed":96842,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247821,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":6090752,"data_stored":7289797,"data_compressed":96842,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313227,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37904,"kb_used_data":6716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333936,"statfs":{"total":96636764160,"available":96597950464,"internally_reserved":0,"allocated":6877184,"data_stored":8971012,"data_compressed":87852,"data_compressed_allocated":122880,"data_compressed_original":245760,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313227,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":4616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":4726784,"data_stored":5319354,"data_compressed":84836,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":6040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":6184960,"data_stored":7482726,"data_compressed":122815,"data_compressed_allocated":167936,"data_compressed_original":765952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280523,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":4468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":4575232,"data_stored":5755812,"data_compressed":93855,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":4876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":4993024,"data_stored":6620087,"data_compressed":93855,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38648,"kb_used_data":7456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333192,"statfs":{"total":96636764160,"available":96597188608,"internally_reserved":0,"allocated":7634944,"data_stored":8964934,"data_compressed":90870,"data_compressed_allocated":126976,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280524,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31216,"kb_used_data":4188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340624,"statfs":{"total":96636764160,"available":96604798976,"internally_reserved":0,"allocated":4288512,"data_stored":4979526,"data_compressed":90870,"data_compressed_allocated":126976,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280523,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":5804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":5943296,"data_stored":6724422,"data_compressed":90870,"data_compressed_allocated":126976,"data_compressed_original":253952,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35972,"kb_used_data":4780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335868,"statfs":{"total":96636764160,"available":96599928832,"internally_reserved":0,"allocated":4894720,"data_stored":5724998,"data_compressed":90870,"data_compressed_allocated":126976,"data_compressed_original":253952,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280523,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34124,"kb_used_data":7096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337716,"statfs":{"total":96636764160,"available":96601821184,"internally_reserved":0,"allocated":7266304,"data_stored":9435974,"data_compressed":90870,"data_compressed_allocated":126976,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":5820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":5959680,"data_stored":7934884,"data_compressed":93855,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280523,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30040,"kb_used_data":3008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341800,"statfs":{"total":96636764160,"available":96606003200,"internally_reserved":0,"allocated":3080192,"data_stored":3441591,"data_compressed":93855,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36468,"kb_used_data":5276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335372,"statfs":{"total":96636764160,"available":96599420928,"internally_reserved":0,"allocated":5402624,"data_stored":6908742,"data_compressed":90870,"data_compressed_allocated":126976,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6713344,"data_stored":9211904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5464064,"data_stored":6873088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5541888,"data_stored":6578176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4395008,"data_stored":6234112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7053312,"data_stored":8589312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3706880,"data_stored":4603904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5361664,"data_stored":6348800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4313088,"data_stored":5349376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":6770688,"data_stored":9146368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4091904,"data_stored":4882432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5365760,"data_stored":7548928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4169728,"data_stored":4964352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2482176,"data_stored":3055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":5357568,"data_stored":6447104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4820992,"data_stored":6533120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3981312,"data_stored":5369856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:07.557 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T04:42:07.558 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-09-16T04:42:07.588 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:07.584+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:07.605 INFO:tasks.rados.rados.0.smithi165.stdout:170: done (3 left) 2024-09-16T04:42:07.606 INFO:tasks.rados.rados.0.smithi165.stdout:173: done (2 left) 2024-09-16T04:42:07.606 INFO:tasks.rados.rados.0.smithi165.stdout:174: done (1 left) 2024-09-16T04:42:07.606 INFO:tasks.rados.rados.0.smithi165.stdout:176: done (0 left) 2024-09-16T04:42:07.606 INFO:tasks.rados.rados.0.smithi165.stdout:177: snap_create 2024-09-16T04:42:07.660 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:07.661 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:07.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:42:07.689 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:07.688+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:07.790 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:07.783+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:07.796 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:07.797 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:07.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:42:07.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:07.891 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:07.888+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:07.934 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:07.934 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:07.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:07.993 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:07.991+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:08.071 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:08.071 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:08.071 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:08.072 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:08.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:08.092 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:08.088+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:08.103 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:08.103 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:08.133 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:08.133 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:08.163 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:08.191 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:08.192 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:08.188+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:08.219 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:08.219 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:08.221 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pg_num to 36 2024-09-16T04:42:08.224 INFO:tasks.rados.rados.0.smithi165.stdout:177: done (0 left) 2024-09-16T04:42:08.224 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 31 current snap is 12 2024-09-16T04:42:08.225 INFO:tasks.rados.rados.0.smithi165.stdout:178: seq_num 83 ranges {1228800=638976,1867776=622592,2490368=147456} 2024-09-16T04:42:08.237 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:42:08.251 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:08.251 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:08.258 INFO:tasks.rados.rados.0.smithi165.stdout:178: writing smithi16530038-31 from 1228800 to 1867776 tid 1 2024-09-16T04:42:08.260 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:08.260 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:08.268 INFO:tasks.rados.rados.0.smithi165.stdout:178: writing smithi16530038-31 from 1867776 to 2490368 tid 2 2024-09-16T04:42:08.271 INFO:tasks.rados.rados.0.smithi165.stdout:178: writing smithi16530038-31 from 2490368 to 2637824 tid 3 2024-09-16T04:42:08.271 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 16 current snap is 12 2024-09-16T04:42:08.271 INFO:tasks.rados.rados.0.smithi165.stdout:179: seq_num 84 ranges {1523712=655360,2179072=638976,2818048=131072} 2024-09-16T04:42:08.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:08.287 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:08.287 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:08.293 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:08.292+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:08.296 INFO:tasks.rados.rados.0.smithi165.stdout:179: writing smithi16530038-16 from 1523712 to 2179072 tid 1 2024-09-16T04:42:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:179: writing smithi16530038-16 from 2179072 to 2818048 tid 2 2024-09-16T04:42:08.304 INFO:tasks.rados.rados.0.smithi165.stdout:179: writing smithi16530038-16 from 2818048 to 2949120 tid 3 2024-09-16T04:42:08.304 INFO:tasks.rados.rados.0.smithi165.stdout:180: copy_from oid 10 from oid 39 current snap is 12 2024-09-16T04:42:08.304 INFO:tasks.rados.rados.0.smithi165.stdout:181: read oid 48 snap 8 2024-09-16T04:42:08.304 INFO:tasks.rados.rados.0.smithi165.stdout:181: expect (ObjNum 54 snap 1 seq_num 54) 2024-09-16T04:42:08.304 INFO:tasks.rados.rados.0.smithi165.stdout:182: copy_from oid 50 from oid 1 current snap is 12 2024-09-16T04:42:08.304 INFO:tasks.rados.rados.0.smithi165.stdout:183: snap_remove snap 7 2024-09-16T04:42:08.307 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:08.329 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:08.354 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:08.354 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:08.374 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:08.374 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:08.392 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:08.394 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:08.394 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:08.394 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:08.392+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:08.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:08.415 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:08.415 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:08.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:08.437 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:08.459 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:08.525 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:08.524+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:08.573 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:08.574 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:08.574 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:08.574 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:08.574 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:08.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:08.596 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:08.592+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:08.697 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:08.692+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:154 3.fs0 3:f6276c08:::smithi16530038-20:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:08.719 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.504080381000001, 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2626387860000001, 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 154 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210502+0000", 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9139999999999999e-06 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210519+0000", 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6969999999999998e-05 2024-09-16T04:42:08.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210555+0000", 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6099000000000001e-05 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210573+0000", 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7770000000000001e-05 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210738+0000", 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016486499999999999 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210962+0000", 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022410600000000001 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.269739+0000", 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.058777538 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272567+0000", 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028276550000000001 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.472907+0000", 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20033962699999999 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.473136+0000", 2024-09-16T04:42:08.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000229242 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:155 3.fs0 3:f6276c08:::smithi16530038-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.503964371999999, 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.061292586, 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.727 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 155 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210614+0000", 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7000000000000003e-07 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210619+0000", 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210636+0000", 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7436999999999999e-05 2024-09-16T04:42:08.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210645+0000", 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.816e-06 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211299+0000", 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065386899999999996 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211385+0000", 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5852000000000004e-05 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271332+0000", 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.05994666 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271584+0000", 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025255399999999999 2024-09-16T04:42:08.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271906+0000", 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032151100000000002 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.734 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:161 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~524288,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896315+0000", 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.818262154999999, 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41137774599999999, 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896315+0000", 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896315+0000", 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896318+0000", 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896331+0000", 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2877000000000001e-05 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896368+0000", 2024-09-16T04:42:08.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7129000000000003e-05 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896395+0000", 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6429999999999999e-05 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.996662+0000", 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100267417 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.996840+0000", 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017823599999999999 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.307693+0000", 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.310852764 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.740 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:162 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~524288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:08.740 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896446+0000", 2024-09-16T04:42:08.740 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.818131296000001, 2024-09-16T04:42:08.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.412080487, 2024-09-16T04:42:08.741 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.741 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.741 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.741 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 162 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896446+0000", 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896446+0000", 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896448+0000", 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896457+0000", 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3670000000000003e-06 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896479+0000", 2024-09-16T04:42:08.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2183000000000001e-05 2024-09-16T04:42:08.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896496+0000", 2024-09-16T04:42:08.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7220000000000001e-05 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.997292+0000", 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10079545600000001 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.997400+0000", 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108319 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.308527+0000", 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31112643899999998 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:163 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~524288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896528+0000", 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.818049176999999, 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41302055999999998, 2024-09-16T04:42:08.746 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 163 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896528+0000", 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896528+0000", 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896530+0000", 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.333e-06 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896539+0000", 2024-09-16T04:42:08.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3829999999999998e-06 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896566+0000", 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6543999999999999e-05 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896585+0000", 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9925999999999999e-05 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.997887+0000", 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101301393 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998027+0000", 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014029600000000001 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.309549+0000", 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31152168499999999 2024-09-16T04:42:08.750 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:169 3.cs0 3:34145aa0:::smithi16530038-26:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.817900755, 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87942165800000005, 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 169 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896679+0000", 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3070000000000001e-06 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896692+0000", 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2785999999999999e-05 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896719+0000", 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7540000000000001e-05 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896735+0000", 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5897e-05 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998807+0000", 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102071833 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998930+0000", 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012282600000000001 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.572766+0000", 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57383636000000005 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574732+0000", 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019661589999999999 2024-09-16T04:42:08.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.775926+0000", 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20119392799999999 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.776098+0000", 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172022 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:170 3.cs0 3:34145aa0:::smithi16530038-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.817814835, 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67734535900000004, 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:08.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896764+0000", 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896768+0000", 2024-09-16T04:42:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896780+0000", 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2003e-05 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896791+0000", 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0876000000000001e-05 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101155+0000", 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20436343900000001 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:08.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101252+0000", 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7195999999999996e-05 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573831+0000", 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.47257916700000002 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573966+0000", 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134816 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574108+0000", 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014204199999999999 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:207 3.fs0 3:f1175b03:::smithi16530038-29:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:08.763 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.614169186000002, 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.69161456600000004, 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 207 2024-09-16T04:42:08.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100411+0000", 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100423+0000", 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.202e-05 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100454+0000", 2024-09-16T04:42:08.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0895999999999997e-05 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100469+0000", 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5614e-05 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174514+0000", 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.074044911000000005 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174748+0000", 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023420200000000001 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.687971+0000", 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51322260399999997 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690498+0000", 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002526978 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791786+0000", 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10128820500000001 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.792023+0000", 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000236772 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:208 3.fs0 3:f1175b03:::smithi16530038-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.614078840000001, 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.58977269499999996, 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.770 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100500+0000", 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100507+0000", 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100524+0000", 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.607e-05 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100532+0000", 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8270000000000004e-06 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274070+0000", 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17353716399999999 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274196+0000", 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012600600000000001 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689716+0000", 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41552020499999998 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689979+0000", 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026336099999999999 2024-09-16T04:42:08.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690271+0000", 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029222200000000002 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:287 3.fs0 3:f1175b03:::smithi16530038-29:3 [read 0~229376,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:54.112480+0000", 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.602097282999999, 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.410377875, 2024-09-16T04:42:08.775 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 287 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112480+0000", 2024-09-16T04:42:08.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112480+0000", 2024-09-16T04:42:08.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112482+0000", 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.313e-06 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112490+0000", 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3170000000000002e-06 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112510+0000", 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0136000000000002e-05 2024-09-16T04:42:08.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112524+0000", 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.434e-05 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112779+0000", 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025434500000000001 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.112788+0000", 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4329999999999993e-06 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.119914+0000", 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007125747 2024-09-16T04:42:08.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.120086+0000", 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172189 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.522858+0000", 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40277205500000002 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.781 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.7.22:2 3.cs0 3:34145aa0:::smithi16530038-26:head [assert-version v13,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.395156242000001, 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61025086299999998, 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.782 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319424+0000", 2024-09-16T04:42:08.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319433+0000", 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4239999999999999e-06 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319460+0000", 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6965999999999999e-05 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319498+0000", 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7518999999999998e-05 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.421237+0000", 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101738922 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.421547+0000", 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000310871 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.518325+0000", 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096777742 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.518478+0000", 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015335199999999999 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.929672+0000", 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41119360100000002 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:356 3.fs0 3:f6276c08:::smithi16530038-20:head [append 0~0] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.457042767000001, 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50700994099999996, 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 356 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257538+0000", 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3170000000000002e-06 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257552+0000", 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.381e-05 2024-09-16T04:42:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257588+0000", 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6046e-05 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357861+0000", 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10027301499999999 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458138+0000", 2024-09-16T04:42:08.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100277112 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458411+0000", 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027305100000000001 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.460377+0000", 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019655160000000001 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465044+0000", 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046678409999999998 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.764419+0000", 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29937459100000002 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.764545+0000", 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125642 2024-09-16T04:42:08.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:357 3.fs0 3:f6276c08:::smithi16530038-20:head [rollback 3] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.356621346000001, 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60733727699999995, 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 357 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357959+0000", 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6630000000000002e-06 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357971+0000", 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2660000000000001e-05 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.358002+0000", 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0793000000000001e-05 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458105+0000", 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100102386 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.462250+0000", 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004145213 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.462405+0000", 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015478599999999999 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463262+0000", 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085764999999999995 2024-09-16T04:42:08.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465233+0000", 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019711049999999999 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.965176+0000", 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.499942461 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.965293+0000", 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011756000000000001 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:358 3.fs0 3:f6276c08:::smithi16530038-20:head [append 0~0] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.256442293999999, 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60770323100000001, 2024-09-16T04:42:08.799 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 358 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458137+0000", 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0229999999999999e-06 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458148+0000", 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0784e-05 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458166+0000", 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8116000000000001e-05 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458182+0000", 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5424e-05 2024-09-16T04:42:08.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463813+0000", 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056313439999999999 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463964+0000", 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105300000000001 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.464741+0000", 2024-09-16T04:42:08.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077701400000000002 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465465+0000", 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072419100000000001 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.065719+0000", 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60025337300000003 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.065838+0000", 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011990899999999999 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:376 3.cs0 3:376489e5:::smithi16530038-34:head [delete] snapc 6=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.446626341, 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.52363831800000005, 2024-09-16T04:42:08.805 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 376 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267954+0000", 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5339999999999998e-06 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267963+0000", 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6700000000000006e-06 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267992+0000", 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8646e-05 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268013+0000", 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0814000000000001e-05 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268212+0000", 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000199681 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268360+0000", 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014787599999999999 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.269584+0000", 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012233210000000001 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.271521+0000", 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373459999999999 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.791438+0000", 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51991722100000004 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.791589+0000", 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151209 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:424 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~573440] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4796452200000001, 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.502816084, 2024-09-16T04:42:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 424 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234936+0000", 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.456e-06 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.235611+0000", 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00067546800000000003 2024-09-16T04:42:08.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.235647+0000", 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5466999999999998e-05 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335792+0000", 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10014563999999999 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335931+0000", 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000138582 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336098+0000", 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167465 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.338085+0000", 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019867890000000001 2024-09-16T04:42:08.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737213+0000", 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39912832100000001 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737613+0000", 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039929100000000002 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737748+0000", 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013560499999999999 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:425 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~425984] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.335847+0000", 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3787307770000004, 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40264106300000002, 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.817 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 425 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335847+0000", 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335847+0000", 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335849+0000", 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336325+0000", 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047620399999999998 2024-09-16T04:42:08.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336349+0000", 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3830000000000001e-05 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336361+0000", 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2356e-05 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.340680+0000", 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043186079999999998 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.340843+0000", 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163499 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.342071+0000", 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012273410000000001 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737290+0000", 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39521969400000001 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.738378+0000", 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001087275 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.738488+0000", 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110203 2024-09-16T04:42:08.822 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:426 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~344064] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3781878379999997, 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90840981600000004, 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.823 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 426 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:08.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336391+0000", 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0699999999999996e-07 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336788+0000", 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039739000000000002 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336805+0000", 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6583999999999999e-05 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336814+0000", 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4299999999999995e-06 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739098+0000", 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40228381099999999 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739254+0000", 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156506 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.740433+0000", 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011782450000000001 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943229+0000", 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20279692899999999 2024-09-16T04:42:08.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244590+0000", 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30136072600000002 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244799+0000", 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020928799999999999 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:427 3.cs0 3:3dc0a373:::smithi16530038-10:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.377746771, 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90837145699999999, 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 427 2024-09-16T04:42:08.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336832+0000", 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7400000000000002e-07 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336837+0000", 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9300000000000002e-06 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336857+0000", 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-05 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336890+0000", 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2852000000000003e-05 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841600+0000", 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50471004399999997 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841814+0000", 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000214589 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.842536+0000", 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072138400000000004 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943303+0000", 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10076771499999999 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245048+0000", 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30174494299999999 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245202+0000", 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015384599999999999 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.834 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:428 3.cs0 3:3dc0a373:::smithi16530038-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3776746649999998, 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1109408999999999, 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 428 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:08.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336904+0000", 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336920+0000", 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6067e-05 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336933+0000", 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336941+0000", 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4600000000000003e-06 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245817+0000", 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90887578899999999 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:08.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.246090+0000", 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000273071 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.447844+0000", 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20175383 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:08.839 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:08.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:42:08.842 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:08.836+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:08.941 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:08.940+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:08.970 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:08.970 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:08.970 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:08.970 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:08.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:42:09.043 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:09.035+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:09.107 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:09.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:42:09.144 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:09.140+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:09.230 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:183: done (5 left) 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:184: delete oid 18 current snap is 12 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:182: finishing copy_from racing read to smithi16530038-50 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 1 to smithi16530038-16 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 2 to smithi16530038-16 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 3 to smithi16530038-16 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 4 to smithi16530038-16 2024-09-16T04:42:09.239 INFO:tasks.rados.rados.0.smithi165.stdout:182: finishing copy_from to smithi16530038-50 2024-09-16T04:42:09.240 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:09.240 INFO:tasks.rados.rados.0.smithi165.stdout:181: expect (ObjNum 54 snap 1 seq_num 54) 2024-09-16T04:42:09.247 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:09.240+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=10, attrs_read=0))", 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.966361237000001, 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30135611200000001, 2024-09-16T04:42:09.254 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284337+0000", 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284378+0000", 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1134000000000003e-05 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.284392+0000", 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3556e-05 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585581+0000", 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30118932599999998 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585693+0000", 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011209600000000001 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=8, attrs_read=0))", 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.288190+0000", 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.962507908999999, 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29726268900000002, 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.258 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288190+0000", 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288190+0000", 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288193+0000", 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2840000000000001e-06 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288223+0000", 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0358999999999999e-05 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288257+0000", 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4236999999999998e-05 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.288279+0000", 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2033000000000001e-05 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585381+0000", 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29710208399999999 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585453+0000", 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1692000000000004e-05 2024-09-16T04:42:09.261 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=10, attrs_read=0))", 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.289624+0000", 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.961073941999999, 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.296969174, 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289624+0000", 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289624+0000", 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289626+0000", 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.756e-06 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289650+0000", 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3734e-05 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289692+0000", 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1811999999999999e-05 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289713+0000", 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1523e-05 2024-09-16T04:42:09.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.586506+0000", 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.296792526 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.586594+0000", 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7823000000000003e-05 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 36/32 ECSubWriteReply(tid=9, last_complete=36'6, committed=1, applied=1))", 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.289945+0000", 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.960753865000001, 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29522546100000002, 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289945+0000", 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289945+0000", 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289946+0000", 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0300000000000001e-06 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289952+0000", 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6100000000000002e-06 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289972+0000", 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.003e-05 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.289983+0000", 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1090000000000001e-05 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585142+0000", 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29515883500000001 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585170+0000", 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7866000000000001e-05 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=8, attrs_read=0))", 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.870517556999999, 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.206073801, 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380181+0000", 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380199+0000", 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8009999999999999e-05 2024-09-16T04:42:09.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380213+0000", 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3966e-05 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380221+0000", 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2169999999999997e-06 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.585758+0000", 2024-09-16T04:42:09.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.205537782 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.586255+0000", 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049623600000000005 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.5s0 36/32 ECSubReadReply(tid=10, attrs_read=0))", 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.380234+0000", 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.870464124000002, 2024-09-16T04:42:09.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27750757599999998, 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380234+0000", 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380234+0000", 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380235+0000", 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7599999999999997e-07 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380248+0000", 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.314e-05 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380258+0000", 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7929999999999992e-06 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.380268+0000", 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9299999999999998e-06 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.657255+0000", 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.276987121 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.657742+0000", 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000487116 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:3 3.5s0 3:a54da884:::smithi16530038-14:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.074238651000002, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50899456499999995, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.277 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176463+0000", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176482+0000", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8975999999999999e-05 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176519+0000", 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6916999999999999e-05 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.276624+0000", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100105124 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478009+0000", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.201384806 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478333+0000", 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000324381 2024-09-16T04:42:09.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.685454+0000", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20712119400000001 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.8s5 39/37 ECSubWrite(tid=6, reqid=client.4728.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.160412507, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.187882937, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090286+0000", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4300000000000003e-07 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090310+0000", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3459999999999999e-05 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090342+0000", 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1973e-05 2024-09-16T04:42:09.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.090369+0000", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7250000000000002e-05 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.177958+0000", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.087588497000000001 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.177971+0000", 2024-09-16T04:42:09.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3869999999999999e-05 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.278092+0000", 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100120098 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:49.278169+0000", 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7245999999999999e-05 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.10s2 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.409445+0000", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.841253297000002, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20474215100000001, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409445+0000", 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409445+0000", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409446+0000", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.384e-06 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409492+0000", 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5126000000000001e-05 2024-09-16T04:42:09.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409533+0000", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0952999999999999e-05 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.409551+0000", 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8666000000000001e-05 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.512744+0000", 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103193255 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.512773+0000", 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8615999999999998e-05 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.614155+0000", 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10138161 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.614187+0000", 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2540999999999999e-05 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.290 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.10s2 42/41 ECSubWrite(tid=33, reqid=client.4728.0:187, at_version=42'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.73521195, 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.203467801, 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515486+0000", 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515487+0000", 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1999999999999998e-07 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515532+0000", 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4295999999999997e-05 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.515554+0000", 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2325999999999999e-05 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.615627+0000", 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100073459 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.715775+0000", 2024-09-16T04:42:09.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10014733300000001 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.715784+0000", 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.718921+0000", 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031364079999999998 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.718954+0000", 2024-09-16T04:42:09.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3612999999999998e-05 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.12s4 42/41 [PushOp(3:4cc0f814:::smithi16530038-19:head, version: 40'6, data_included: [0~225280], data_size: 225280, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:4cc0f814:::smithi16530038-19:head@40'6, size: 901120, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 901120, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:52.932709+0000", 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.317989289, 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.202621732, 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.932709+0000", 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.932709+0000", 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.932710+0000", 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.540000000000001e-07 2024-09-16T04:42:09.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.933346+0000", 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063553100000000005 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:52.933371+0000", 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5754000000000001e-05 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.033558+0000", 2024-09-16T04:42:09.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100186579 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.133836+0000", 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100278226 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.135331+0000", 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014946880000000001 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.3s3 43/37 ECSubRead(tid=175, to_read={3:c31e3ec6:::smithi16530038-50:head=0,208896,0}, subchunks={3:c31e3ec6:::smithi16530038-50:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.306734+0000", 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.943964755, 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15662287999999999, 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.306734+0000", 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.306734+0000", 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.306736+0000", 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.306755+0000", 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9904000000000001e-05 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.306806+0000", 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0269999999999998e-05 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.306835+0000", 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9519e-05 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.458839+0000", 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.152004258 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.463356+0000", 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045170089999999998 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.4.22:2 3.5s0 3:a54da884:::smithi16530038-14:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:54.215342+0000", 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.03535647, 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11581424999999999, 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.215342+0000", 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.215342+0000", 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.215345+0000", 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8700000000000001e-06 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.215355+0000", 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0293e-05 2024-09-16T04:42:09.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.215377+0000", 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1444e-05 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.215393+0000", 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6293000000000001e-05 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.226530+0000", 2024-09-16T04:42:09.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011137634 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.226680+0000", 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000149209 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.331156+0000", 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104476507 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.12s4 44/41 ECSubWrite(tid=180, reqid=osd.13.0:179, at_version=0'0, trim_to=0'0, roll_forward_to=44'10))", 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:54.430110+0000", 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.820588597, 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20130389000000001, 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.430110+0000", 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.430110+0000", 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.430111+0000", 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.124e-06 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.430132+0000", 2024-09-16T04:42:09.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0856000000000002e-05 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.430167+0000", 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5216000000000003e-05 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.530273+0000", 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100105598 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.630491+0000", 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100218259 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.630507+0000", 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6137000000000002e-05 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.631347+0000", 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084023699999999997 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.631414+0000", 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6463000000000004e-05 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:359 3.5s0 3:a54da884:::smithi16530038-14:head [read 0~229376,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257726+0000", 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.992972324, 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30495125899999997, 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 359 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257726+0000", 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257726+0000", 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257730+0000", 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6160000000000002e-06 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257749+0000", 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.942e-05 2024-09-16T04:42:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257782+0000", 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2772999999999999e-05 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257798+0000", 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6626999999999999e-05 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.349728+0000", 2024-09-16T04:42:09.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.091929364 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.349891+0000", 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016355499999999999 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.562677+0000", 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.212785904 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.315 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:360 3.5s0 3:a54da884:::smithi16530038-14:head [read 0~229376,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257826+0000", 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.992872175, 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30567145899999998, 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.316 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 360 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257826+0000", 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257826+0000", 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257827+0000", 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.47e-07 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257833+0000", 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.246e-06 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257846+0000", 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2787e-05 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257855+0000", 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0459999999999994e-06 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.350510+0000", 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092655111999999998 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.350754+0000", 2024-09-16T04:42:09.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024403900000000001 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.563498+0000", 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21274328200000001 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:361 3.5s0 3:a54da884:::smithi16530038-14:head [read 0~229376,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.992830262, 2024-09-16T04:42:09.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.306317012, 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 361 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257869+0000", 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257874+0000", 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.023e-06 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257885+0000", 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257892+0000", 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5630000000000003e-06 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.351201+0000", 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093309317000000003 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.351321+0000", 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119459 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.564185+0000", 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21286419300000001 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.324 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:392 3.5s0 3:a54da884:::smithi16530038-14:head [copy-from ver 8] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.1082955939999994, 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.36993664399999998, 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:09.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 392 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142404+0000", 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.674e-06 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142411+0000", 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0400000000000004e-06 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142438+0000", 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6486e-05 2024-09-16T04:42:09.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142449+0000", 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0835999999999999e-05 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142539+0000", 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9823999999999999e-05 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142675+0000", 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013676899999999999 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.294617+0000", 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15194110199999999 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.308789+0000", 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014172736999999999 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.512242+0000", 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20345274799999999 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.512339+0000", 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7428000000000003e-05 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:393 3.5s0 3:a54da884:::smithi16530038-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:00.142476+0000", 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.1082224919999994, 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.22230703299999999, 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 393 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142476+0000", 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142476+0000", 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142477+0000", 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3300000000000002e-07 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142480+0000", 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0010000000000002e-06 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142489+0000", 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6080000000000002e-06 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142496+0000", 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1110000000000002e-06 2024-09-16T04:42:09.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142937+0000", 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044046500000000002 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.143050+0000", 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113136 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.364244+0000", 2024-09-16T04:42:09.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.221193907 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.364491+0000", 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024703399999999998 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.364783+0000", 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000292138 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.9s4 50/47 ECSubWrite(tid=57, reqid=client.4728.0:406, at_version=50'30, trim_to=0'0, roll_forward_to=48'26))", 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:00.263668+0000", 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.9870300580000002, 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.198696487, 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.263668+0000", 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.263668+0000", 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.263669+0000", 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1100000000000005e-07 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.263686+0000", 2024-09-16T04:42:09.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7037e-05 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.263699+0000", 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3010000000000001e-05 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.263708+0000", 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6989999999999992e-06 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.461605+0000", 2024-09-16T04:42:09.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19789706300000001 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.461616+0000", 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1353e-05 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.462322+0000", 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070585099999999996 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:09.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.462365+0000", 2024-09-16T04:42:09.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2663000000000001e-05 2024-09-16T04:42:09.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:09.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:09.340 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:09.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:09.342 INFO:tasks.rados.rados.0.smithi165.stdout:178: finishing write tid 1 to smithi16530038-31 2024-09-16T04:42:09.342 INFO:tasks.rados.rados.0.smithi165.stdout:178: finishing write tid 2 to smithi16530038-31 2024-09-16T04:42:09.342 INFO:tasks.rados.rados.0.smithi165.stdout:178: finishing write tid 3 to smithi16530038-31 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:180: finishing copy_from racing read to smithi16530038-10 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:178: finishing write tid 4 to smithi16530038-31 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 5 to smithi16530038-16 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 16 updating version 0 to 27 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 16 updating version 27 to 28 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 16 version 28 is already newer than 26 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 16 updating version 28 to 29 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 29 (ObjNum 84 snap 12 seq_num 84) dirty exists 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:179: left oid 16 (ObjNum 84 snap 12 seq_num 84) 2024-09-16T04:42:09.343 INFO:tasks.rados.rados.0.smithi165.stdout:178: finishing write tid 5 to smithi16530038-31 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:178: oid 31 updating version 0 to 25 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:178: oid 31 updating version 25 to 27 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:178: oid 31 version 27 is already newer than 24 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:178: oid 31 version 27 is already newer than 26 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 27 (ObjNum 83 snap 12 seq_num 83) dirty exists 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:178: left oid 31 (ObjNum 83 snap 12 seq_num 83) 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:180: finishing copy_from to smithi16530038-10 2024-09-16T04:42:09.344 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T04:42:09.386 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:09.384+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:09.446 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:09.444+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:09.454 INFO:tasks.rados.rados.0.smithi165.stdout:178: done (5 left) 2024-09-16T04:42:09.454 INFO:tasks.rados.rados.0.smithi165.stdout:179: done (4 left) 2024-09-16T04:42:09.454 INFO:tasks.rados.rados.0.smithi165.stdout:180: done (3 left) 2024-09-16T04:42:09.454 INFO:tasks.rados.rados.0.smithi165.stdout:181: done (2 left) 2024-09-16T04:42:09.454 INFO:tasks.rados.rados.0.smithi165.stdout:182: done (1 left) 2024-09-16T04:42:09.454 INFO:tasks.rados.rados.0.smithi165.stdout:184: done (0 left) 2024-09-16T04:42:09.454 INFO:tasks.rados.rados.0.smithi165.stdout:185: snap_remove snap 10 2024-09-16T04:42:09.522 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:09.522 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:09.522 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:09.522 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:09.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:09.581 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:09.580+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:09.648 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:09.644+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:09.713 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:09.713 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:09.713 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:09.713 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:09.713 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:09.727 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:09.749 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:09.744+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:09.850 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:09.847+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:09.899 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:09.899 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:09.899 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:09.899 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s4 36/32 ECSubWrite(tid=8, reqid=client.4728.0:109, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.521613323, 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201785732, 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.900 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374507+0000", 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374649+0000", 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014185599999999999 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374672+0000", 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3277000000000001e-05 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374685+0000", 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2797e-05 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574940+0000", 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20025501600000001 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574956+0000", 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5936000000000001e-05 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576258+0000", 2024-09-16T04:42:09.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001302388 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576291+0000", 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 36/32 ECSubWrite(tid=4, reqid=client.4728.0:112, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.322506996000001, 2024-09-16T04:42:09.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20173359499999999, 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:09.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573614+0000", 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.114e-06 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574099+0000", 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048458300000000002 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574141+0000", 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2551e-05 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574168+0000", 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6899000000000001e-05 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674350+0000", 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100181464 2024-09-16T04:42:09.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674362+0000", 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2138e-05 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775305+0000", 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100943171 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775345+0000", 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0674999999999999e-05 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:164 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.998584562000001, 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40676020400000001, 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.909 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 164 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:09.910 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897538+0000", 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0500000000000002e-06 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897554+0000", 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5886e-05 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897591+0000", 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6899e-05 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897617+0000", 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5922e-05 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897727+0000", 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011011200000000001 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897835+0000", 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107889 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304263+0000", 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40642834999999999 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304295+0000", 2024-09-16T04:42:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1096000000000002e-05 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:165 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.998444874, 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20425574899999999, 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.914 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 165 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:09.915 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897676+0000", 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897685+0000", 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.183e-06 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897709+0000", 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4216e-05 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997840+0000", 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131417 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098015+0000", 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100174894 2024-09-16T04:42:09.917 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098124+0000", 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108714 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101885+0000", 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037611459999999999 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101930+0000", 2024-09-16T04:42:09.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4675999999999997e-05 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:166 3.6s0 3.a17ee16 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.898228611, 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.306539124, 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.919 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 166 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.920 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997892+0000", 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.921 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997905+0000", 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2583e-05 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997926+0000", 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.101e-05 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098022+0000", 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100096478 2024-09-16T04:42:09.922 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304408+0000", 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20638583099999999 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304429+0000", 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1132e-05 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.923 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.cs2 36/32 ECSubRead(tid=30, to_read={3:372a9655:::smithi16530038-32:head=0,131072,0}, subchunks={3:372a9655:::smithi16530038-32:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:47.098249+0000", 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.797869656, 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200837185, 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098249+0000", 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098249+0000", 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098250+0000", 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4300000000000003e-07 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098256+0000", 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3899999999999998e-06 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098270+0000", 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3766e-05 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.198426+0000", 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100155832 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.298779+0000", 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10035303600000001 2024-09-16T04:42:09.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.299086+0000", 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00030761800000000001 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.16s0 42/41 ECSubRead(tid=6, to_read={3:6877e850:::smithi16530038-5:head=0,20480,0}, subchunks={3:6877e850:::smithi16530038-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:09.928 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.587895544999999, 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20348143299999999, 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:09.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.930 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308260+0000", 2024-09-16T04:42:09.931 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6318e-05 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308312+0000", 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2611999999999997e-05 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511485+0000", 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20317243800000001 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511705+0000", 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220065 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.933 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.587219712, 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20303517200000001, 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308901+0000", 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308970+0000", 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9240999999999997e-05 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309005+0000", 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4314999999999999e-05 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309023+0000", 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8879999999999999e-05 2024-09-16T04:42:09.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511826+0000", 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202802133 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511934+0000", 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108756 2024-09-16T04:42:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.586984919999999, 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20302847199999999, 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309135+0000", 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:09.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309242+0000", 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106959 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309260+0000", 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309270+0000", 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0339e-05 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512036+0000", 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20276591199999999 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512162+0000", 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126675 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.941 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.14s4 43/41 ECSubWrite(tid=30, reqid=client.4728.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 16.556842518, 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20195774699999999, 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339277+0000", 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.61e-07 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.943 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339289+0000", 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1533e-05 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339303+0000", 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4002000000000001e-05 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439391+0000", 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10008851100000001 2024-09-16T04:42:09.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439604+0000", 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301100000000001 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439617+0000", 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3076e-05 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541138+0000", 2024-09-16T04:42:09.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10152072299999999 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541234+0000", 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6130000000000003e-05 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.9s2 48/47 ECSubWrite(tid=51, reqid=client.4728.0:309, at_version=48'24, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:58.360798+0000", 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.535321286, 2024-09-16T04:42:09.946 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20098608500000001, 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.360798+0000", 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.360798+0000", 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.360799+0000", 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.313e-06 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.360829+0000", 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0378999999999998e-05 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.360858+0000", 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8419999999999999e-05 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.360883+0000", 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4811999999999999e-05 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.560596+0000", 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.199713103 2024-09-16T04:42:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.560628+0000", 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2393e-05 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.561684+0000", 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001056392 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.561784+0000", 2024-09-16T04:42:09.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9272999999999996e-05 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.14s4,RELEASE,e:48,from:8(0),reservation_nonce:0)", 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:58.556646+0000", 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.339473151, 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20048906, 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.951 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.556646+0000", 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.556646+0000", 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.556647+0000", 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4130000000000001e-06 2024-09-16T04:42:09.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.556656+0000", 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2860000000000005e-06 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.556699+0000", 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2225999999999999e-05 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.556718+0000", 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9426e-05 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.757066+0000", 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20034806399999999 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.757105+0000", 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8618999999999997e-05 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.757135+0000", 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0026e-05 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/32 ECSubWrite(tid=82, reqid=osd.11.0:80, at_version=0'0, trim_to=0'0, roll_forward_to=49'27))", 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.763729+0000", 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.132390331, 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20062812099999999, 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.763729+0000", 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.763729+0000", 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.763731+0000", 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9530000000000002e-06 2024-09-16T04:42:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.763753+0000", 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2535999999999999e-05 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.763786+0000", 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3259000000000001e-05 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.863905+0000", 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011897 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.864631+0000", 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072527300000000002 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.864655+0000", 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4793000000000001e-05 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.964294+0000", 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.099639156000000006 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.964357+0000", 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2181000000000007e-05 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 53/32 ECSubWrite(tid=89, reqid=client.4728.0:426, at_version=53'29, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9619245010000004, 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20199651099999999, 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934195+0000", 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934336+0000", 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014033399999999999 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934357+0000", 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1769000000000001e-05 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034519+0000", 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016164599999999 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034685+0000", 2024-09-16T04:42:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000166312 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034712+0000", 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6959999999999999e-05 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136138+0000", 2024-09-16T04:42:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101425772 2024-09-16T04:42:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136191+0000", 2024-09-16T04:42:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2868000000000003e-05 2024-09-16T04:42:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=115, reqid=client.4728.0:457, at_version=58'26, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.596763342, 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.240164188, 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5699999999999999e-07 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299490+0000", 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013428299999999999 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299510+0000", 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437999999999999e-05 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299523+0000", 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3132000000000001e-05 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436652+0000", 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13712892300000001 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436659+0000", 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4410000000000004e-06 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539460+0000", 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102800512 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539520+0000", 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0201999999999997e-05 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=118, reqid=client.4728.0:458, at_version=58'27, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.405307+0000", 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.4908123049999999, 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.134329438, 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405307+0000", 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405307+0000", 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405308+0000", 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7630000000000001e-06 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405431+0000", 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012248000000000001 2024-09-16T04:42:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405453+0000", 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2327000000000002e-05 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405466+0000", 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2661e-05 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.437561+0000", 2024-09-16T04:42:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.032095349000000002 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.437574+0000", 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3169999999999999e-05 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539553+0000", 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10197906299999999 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539636+0000", 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2625000000000003e-05 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=119, reqid=client.4728.0:459, at_version=58'28, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.406619+0000", 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.4895003600000001, 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13309214799999999, 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.406619+0000", 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.406619+0000", 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.406619+0000", 2024-09-16T04:42:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.406669+0000", 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9675000000000001e-05 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.406683+0000", 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3573e-05 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.406693+0000", 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9839999999999996e-06 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.538257+0000", 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13156428000000001 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.538289+0000", 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2236000000000001e-05 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539642+0000", 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013525410000000001 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539711+0000", 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8972000000000003e-05 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=120, reqid=client.4728.0:460, at_version=58'29, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.407566+0000", 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.4885530069999999, 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13231710799999999, 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407566+0000", 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407566+0000", 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407567+0000", 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5700000000000002e-07 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407596+0000", 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9658e-05 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407610+0000", 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4225e-05 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407619+0000", 2024-09-16T04:42:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0520000000000007e-06 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539156+0000", 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13153680000000001 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539172+0000", 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5352000000000001e-05 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539839+0000", 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00066713099999999995 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539883+0000", 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=122, reqid=client.4728.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=58'29))", 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.0836375309999999, 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20212734700000001, 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812484+0000", 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3700000000000002e-06 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812511+0000", 2024-09-16T04:42:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.775e-05 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812550+0000", 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812567+0000", 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7186000000000001e-05 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812770+0000", 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202862 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812799+0000", 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8716000000000001e-05 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014571+0000", 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201771586 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014609+0000", 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8124999999999998e-05 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.bs1 58/32 ECSubRead(tid=123, to_read={3:d1897791:::smithi16530038-16:head=0,4096,0}, subchunks={3:d1897791:::smithi16530038-16:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.813815+0000", 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.0823035219999999, 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200748223, 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813815+0000", 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813815+0000", 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813817+0000", 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9599999999999999e-06 2024-09-16T04:42:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813839+0000", 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1979000000000001e-05 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813875+0000", 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5366e-05 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813909+0000", 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4459000000000003e-05 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014405+0000", 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20049578300000001 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014564+0000", 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015867600000000001 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:09.996 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:09.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:09.998 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:09.995+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:10.099 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:10.096+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:10.128 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:10.128 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:10.128 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:10.128 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:10.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:10.200 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:10.199+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:10.235 INFO:tasks.rados.rados.0.smithi165.stdout:185: done (0 left) 2024-09-16T04:42:10.235 INFO:tasks.rados.rados.0.smithi165.stdout:186: snap_create 2024-09-16T04:42:10.301 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:10.295+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:10.313 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:10.313 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:10.313 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:10.313 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:10.313 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:10.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:10.402 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:10.395+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:10.455 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:137 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.390577999000001, 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50711364199999998, 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 137 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062314+0000", 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062318+0000", 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1710000000000004e-06 2024-09-16T04:42:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062327+0000", 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0089999999999996e-06 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162444+0000", 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011658 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163173+0000", 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072917800000000003 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163440+0000", 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026697899999999998 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164004+0000", 2024-09-16T04:42:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056357499999999997 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164775+0000", 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00077136499999999998 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569249+0000", 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.404473996 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569427+0000", 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177906 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:138 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.290409115999999, 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.71018989099999996, 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.462 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 138 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.463 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162484+0000", 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162491+0000", 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162507+0000", 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5141e-05 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162514+0000", 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8460000000000008e-06 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165370+0000", 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028555719999999998 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165438+0000", 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8466999999999993e-05 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570190+0000", 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40475178899999997 2024-09-16T04:42:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570271+0000", 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0437000000000003e-05 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.872672+0000", 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30240151100000001 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.467 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:181 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.360535850000002, 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.514197601, 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.468 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 181 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092358+0000", 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9500000000000001e-06 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093295+0000", 2024-09-16T04:42:10.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00093678900000000005 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093330+0000", 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5039000000000001e-05 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093348+0000", 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7210999999999998e-05 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.471 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269509+0000", 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17616195900000001 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269804+0000", 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029419599999999999 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272744+0000", 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029401789999999998 2024-09-16T04:42:10.472 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279866+0000", 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007122499 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606426+0000", 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32655929 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606553+0000", 2024-09-16T04:42:10.473 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000127489 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:182 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.354207405, 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50813697599999996, 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.474 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 182 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.475 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0260000000000002e-06 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099161+0000", 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000474556 2024-09-16T04:42:10.476 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099192+0000", 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1612000000000002e-05 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099207+0000", 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4796e-05 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275225+0000", 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17601765799999999 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275471+0000", 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024600099999999998 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:42:10.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001437422 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280012+0000", 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031043899999999998 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606726+0000", 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32671404399999998 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.479 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606821+0000", 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4470999999999997e-05 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:183 3.bs0 3:d388eed4:::smithi16530038-28:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.353656996000002, 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60643071599999998, 2024-09-16T04:42:10.480 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 183 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099235+0000", 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.017e-06 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099242+0000", 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4230000000000002e-06 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099256+0000", 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3936e-05 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099264+0000", 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278076+0000", 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17881208600000001 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278318+0000", 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024238100000000001 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.484 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279368+0000", 2024-09-16T04:42:10.485 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010492209999999999 2024-09-16T04:42:10.485 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.485 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281553+0000", 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021853739999999999 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705572+0000", 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.42401889100000001 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705665+0000", 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3173999999999999e-05 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:191 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.351796840999999, 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68460226999999996, 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.487 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 191 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101096+0000", 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-06 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101104+0000", 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1240000000000005e-06 2024-09-16T04:42:10.489 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101122+0000", 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8045000000000001e-05 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101130+0000", 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2670000000000006e-06 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280859+0000", 2024-09-16T04:42:10.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179728363 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280864+0000", 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7370000000000003e-06 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.405994+0000", 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.12512986500000001 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.406243+0000", 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024852700000000001 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.785697+0000", 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.37945393900000002 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.492 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:192 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.351744622999998, 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68518017200000003, 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.493 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 192 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101154+0000", 2024-09-16T04:42:10.495 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3300000000000004e-06 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101160+0000", 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.348e-06 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101164+0000", 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2899999999999996e-06 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281071+0000", 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179906749 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281074+0000", 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.976e-06 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.497 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506776+0000", 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22570168299999999 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506893+0000", 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011752300000000001 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.498 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786327+0000", 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27943377000000003 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:199 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.351670263999999, 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50411928699999997, 2024-09-16T04:42:10.499 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 199 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.500 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101222+0000", 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3799999999999997e-07 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101227+0000", 2024-09-16T04:42:10.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8559999999999998e-06 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101240+0000", 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3485e-05 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101248+0000", 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3699999999999995e-06 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.502 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280982+0000", 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17973399200000001 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281111+0000", 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012887099999999999 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.605340+0000", 2024-09-16T04:42:10.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32422917499999998 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:200 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.35163339, 2024-09-16T04:42:10.504 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50471661899999998, 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 200 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7300000000000001e-07 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101262+0000", 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.14e-06 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101269+0000", 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6399999999999997e-06 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101275+0000", 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6910000000000002e-06 2024-09-16T04:42:10.507 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281623+0000", 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18034850899999999 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281792+0000", 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016865700000000001 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.605975+0000", 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32418250900000001 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:201 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.351611536, 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68324074999999995, 2024-09-16T04:42:10.509 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 201 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.510 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101282+0000", 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9750000000000001e-06 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101287+0000", 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.651e-06 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.201419+0000", 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10013264099999999 2024-09-16T04:42:10.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482351+0000", 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28093166899999999 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482482+0000", 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013094199999999999 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.513 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.784521+0000", 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30203853899999999 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:354 3.17s0 3:ed82f61a:::smithi16530038-1:3 [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:58.147804+0000", 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.305087482999999, 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.41067514300000002, 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.514 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 354 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147804+0000", 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147804+0000", 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147805+0000", 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5e-06 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147817+0000", 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1253000000000001e-05 2024-09-16T04:42:10.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147843+0000", 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6055999999999999e-05 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147858+0000", 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4949999999999999e-05 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.148045+0000", 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000187837 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.148055+0000", 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1670000000000005e-06 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.156045+0000", 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079905459999999994 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.518 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.156173+0000", 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012744300000000001 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.558479+0000", 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40230639099999999 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.519 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:372 3.1s0 3:8583b9f2:::smithi16530038-39:3 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.271609238, 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.58438020099999999, 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 372 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.520 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181283+0000", 2024-09-16T04:42:10.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1100000000000005e-07 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181288+0000", 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4739999999999996e-06 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181300+0000", 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1266e-05 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181308+0000", 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5669999999999995e-06 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181619+0000", 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031025600000000002 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181627+0000", 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3329999999999997e-06 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288322+0000", 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669511499999999 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288524+0000", 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202511 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.765662+0000", 2024-09-16T04:42:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.47713786800000002 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:452 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.1939373820000001, 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57389822599999996, 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 452 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258957+0000", 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9510000000000002e-06 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260061+0000", 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011037410000000001 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260099+0000", 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8401e-05 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260123+0000", 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3808999999999999e-05 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260226+0000", 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010314000000000001 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260310+0000", 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3652000000000005e-05 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.261217+0000", 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090691199999999995 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.262686+0000", 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014689 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832721+0000", 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57003547600000004 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832852+0000", 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013124400000000001 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:453 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~622592] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.1832595079999999, 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56351327799999995, 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 453 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269634+0000", 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2069999999999998e-06 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270168+0000", 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053375500000000002 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270185+0000", 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7231000000000002e-05 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270194+0000", 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4349999999999997e-06 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270266+0000", 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2702999999999998e-05 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270366+0000", 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9485e-05 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271233+0000", 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00086726900000000005 2024-09-16T04:42:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374033+0000", 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10280007300000001 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833038+0000", 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45900526899999999 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833145+0000", 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106851 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:454 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~147456] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.1809947279999999, 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56224230100000006, 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 454 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271898+0000", 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.364e-06 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272075+0000", 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017664100000000001 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272088+0000", 2024-09-16T04:42:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3192e-05 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272097+0000", 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4720000000000001e-06 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372245+0000", 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10014781 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372332+0000", 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7258999999999999e-05 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372854+0000", 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052174600000000002 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374728+0000", 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018736219999999999 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834034+0000", 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45930626099999999 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834139+0000", 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000104934 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:455 3.1s0 3:803446a7:::smithi16530038-31:head [setxattr _header (57) in=7b] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.1807703649999999, 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.66155386500000002, 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 455 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272122+0000", 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272127+0000", 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5369999999999997e-06 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272135+0000", 2024-09-16T04:42:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272141+0000", 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5999999999999997e-06 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373376+0000", 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101234779 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373445+0000", 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9507999999999999e-05 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373742+0000", 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029695500000000001 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374731+0000", 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098912400000000008 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933530+0000", 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.55879828600000003 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933675+0000", 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145353 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:456 3.1s0 3:803446a7:::smithi16530038-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.1807391819999999, 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.76406687699999998, 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 456 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272153+0000", 2024-09-16T04:42:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0100000000000005e-07 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272155+0000", 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.638e-06 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272165+0000", 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1579999999999994e-06 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272171+0000", 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373963+0000", 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10179178799999999 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374024+0000", 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0791999999999997e-05 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934462+0000", 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56043857600000002 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934706+0000", 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024387000000000001 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.036219+0000", 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10151300000000001 2024-09-16T04:42:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:457 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~655360] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.297257+0000", 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.1556346799999999, 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.412841456, 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 457 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297257+0000", 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297257+0000", 2024-09-16T04:42:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297259+0000", 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.979e-06 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297936+0000", 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00067768099999999996 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297958+0000", 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1838e-05 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297969+0000", 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0669000000000001e-05 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.298054+0000", 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5420999999999994e-05 2024-09-16T04:42:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.298187+0000", 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013230499999999999 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011698640000000001 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.300800+0000", 2024-09-16T04:42:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014438459999999999 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.709969+0000", 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40916917000000003 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.710098+0000", 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012868299999999999 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:458 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.149470118, 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50818626099999997, 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 458 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303426+0000", 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5870000000000003e-06 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303961+0000", 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053495599999999995 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303975+0000", 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3835999999999999e-05 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404043+0000", 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100068845 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404113+0000", 2024-09-16T04:42:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9350999999999994e-05 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404242+0000", 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012877400000000001 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405312+0000", 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001070812 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407819+0000", 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002506131 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811485+0000", 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40366653800000002 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811608+0000", 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012243099999999999 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:461 3.bs0 3:d1897791:::smithi16530038-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.0487374159999998, 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.61182177000000004, 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 461 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6399999999999998e-07 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404155+0000", 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2550000000000001e-06 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404160+0000", 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0890000000000002e-06 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404162+0000", 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8399999999999999e-06 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507951+0000", 2024-09-16T04:42:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103788373 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.508127+0000", 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017614900000000001 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813061+0000", 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30493453100000001 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813333+0000", 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027161699999999998 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.015976+0000", 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20264265200000001 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:10.572 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:10.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:10.574 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:10.575 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:10.572+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:10.675 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:10.671+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:10.748 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:10.748 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:10.748 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:10.748 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:10.764 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:42:10.774 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:10.774 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:10.776 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:10.772+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:10.796 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:10.796 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:10.817 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:10.817 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:10.838 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:10.838 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:10.856 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:10.876 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:10.872+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:10.882 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:10.882 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:10.882 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:10.882 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:10.882 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:10.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:42:10.918 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:10.918 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:10.945 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:10.945 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:10.971 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:10.971 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:10.977 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:10.971+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:10.995 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:10.995 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:11.020 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:11.044 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:11.045 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:11.047 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:11.047 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:11.062 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:42:11.069 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:11.070 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:11.078 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:11.076+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:11.087 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:11.087 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:11.108 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:11.108 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:11.129 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:11.150 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:11.150 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:11.179 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:11.171+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:11.195 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:11.195 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:11.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:42:11.237 INFO:tasks.rados.rados.0.smithi165.stdout:186: done (0 left) 2024-09-16T04:42:11.237 INFO:tasks.rados.rados.0.smithi165.stdout:187: read oid 46 snap -1 2024-09-16T04:42:11.238 INFO:tasks.rados.rados.0.smithi165.stdout:187: expect (ObjNum 60 snap 3 seq_num 60) 2024-09-16T04:42:11.238 INFO:tasks.rados.rados.0.smithi165.stdout:188: read oid 7 snap 9 2024-09-16T04:42:11.238 INFO:tasks.rados.rados.0.smithi165.stdout:188: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:11.238 INFO:tasks.rados.rados.0.smithi165.stdout:189: snap_create 2024-09-16T04:42:11.297 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:11.296+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:11.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:42:11.375 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:11.376 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:11.380 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:11.376+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:11.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:42:11.481 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:11.475+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:11.564 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:11.564 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:11.579 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:42:11.582 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:11.580+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:11.683 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:11.680+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:11.709 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:11.710 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:11.723 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:42:11.759 INFO:tasks.rados.rados.0.smithi165.stdout:187: expect (ObjNum 60 snap 3 seq_num 60) 2024-09-16T04:42:11.785 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:11.779+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:11.842 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:11.842 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:11.856 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:42:11.871 INFO:tasks.rados.rados.0.smithi165.stdout:188: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:11.887 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:11.883+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:11.987 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:11.983+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:12.022 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:12.022 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:12.036 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:42:12.127 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:12.123+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:12.189 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:12.188+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:12.204 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:12.204 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:12.219 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:42:12.237 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:42:12.242 INFO:tasks.rados.rados.0.smithi165.stdout:187: done (2 left) 2024-09-16T04:42:12.242 INFO:tasks.rados.rados.0.smithi165.stdout:188: done (1 left) 2024-09-16T04:42:12.243 INFO:tasks.rados.rados.0.smithi165.stdout:189: done (0 left) 2024-09-16T04:42:12.243 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 13 current snap is 14 2024-09-16T04:42:12.243 INFO:tasks.rados.rados.0.smithi165.stdout:190: seq_num 86 ranges {212992=688128,901120=540672,1441792=229376} 2024-09-16T04:42:12.257 INFO:tasks.rados.rados.0.smithi165.stdout:190: writing smithi16530038-13 from 212992 to 901120 tid 1 2024-09-16T04:42:12.266 INFO:tasks.rados.rados.0.smithi165.stdout:190: writing smithi16530038-13 from 901120 to 1441792 tid 2 2024-09-16T04:42:12.270 INFO:tasks.rados.rados.0.smithi165.stdout:190: writing smithi16530038-13 from 1441792 to 1671168 tid 3 2024-09-16T04:42:12.270 INFO:tasks.rados.rados.0.smithi165.stdout:191: copy_from oid 15 from oid 5 current snap is 14 2024-09-16T04:42:12.270 INFO:tasks.rados.rados.0.smithi165.stdout:192: rollback oid 19 current snap is 14 2024-09-16T04:42:12.270 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 19 to 8 2024-09-16T04:42:12.270 INFO:tasks.rados.rados.0.smithi165.stdout:193: copy_from oid 21 from oid 24 current snap is 14 2024-09-16T04:42:12.270 INFO:tasks.rados.rados.0.smithi165.stdout:194: copy_from oid 3 from oid 7 current snap is 14 2024-09-16T04:42:12.271 INFO:tasks.rados.rados.0.smithi165.stdout:195: read oid 33 snap -1 2024-09-16T04:42:12.271 INFO:tasks.rados.rados.0.smithi165.stdout:195: expect (ObjNum 75 snap 6 seq_num 75) 2024-09-16T04:42:12.271 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 16 current snap is 14 2024-09-16T04:42:12.271 INFO:tasks.rados.rados.0.smithi165.stdout:196: seq_num 87 ranges {2949120=704512,3653632=688128,4341760=81920} 2024-09-16T04:42:12.290 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:12.288+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:12.318 INFO:tasks.rados.rados.0.smithi165.stdout:196: writing smithi16530038-16 from 2949120 to 3653632 tid 1 2024-09-16T04:42:12.324 INFO:tasks.rados.rados.0.smithi165.stdout:196: writing smithi16530038-16 from 3653632 to 4341760 tid 2 2024-09-16T04:42:12.325 INFO:tasks.rados.rados.0.smithi165.stdout:196: writing smithi16530038-16 from 4341760 to 4423680 tid 3 2024-09-16T04:42:12.325 INFO:tasks.rados.rados.0.smithi165.stdout:197: rmattr oid 34 current snap is 14 2024-09-16T04:42:12.325 INFO:tasks.rados.rados.0.smithi165.stdout:197: done (7 left) 2024-09-16T04:42:12.325 INFO:tasks.rados.rados.0.smithi165.stdout:198: snap_remove snap 12 2024-09-16T04:42:12.392 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:12.388+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:12.392 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:12.392 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:12.407 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:42:12.492 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:12.487+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:12.529 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:12.530 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:12.544 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:42:12.593 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:12.587+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:12.667 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:12.667 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:12.682 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:42:12.723 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:12.719+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:12.794 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:12.787+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:12.852 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:12.852 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:12.866 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:42:12.896 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:12.892+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:12.996 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:12.992+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:13.034 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:13.034 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:13.048 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:42:13.097 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:13.091+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:13.169 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:13.175 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:13.175 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:13.190 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:42:13.213 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:13.212+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:13.248 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:42:13.249 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:42:13.249 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T04:42:13.249 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T04:42:13.267 INFO:tasks.rados.rados.0.smithi165.stdout:198: done (7 left) 2024-09-16T04:42:13.267 INFO:tasks.rados.rados.0.smithi165.stdout:199: read oid 39 snap -1 2024-09-16T04:42:13.267 INFO:tasks.rados.rados.0.smithi165.stdout:199: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:13.267 INFO:tasks.rados.rados.0.smithi165.stdout:200: rmattr oid 4 current snap is 14 2024-09-16T04:42:13.267 INFO:tasks.rados.rados.0.smithi165.stdout:192: finishing rollback tid 0 to smithi16530038-19 2024-09-16T04:42:13.267 INFO:tasks.rados.rados.0.smithi165.stdout:196: finishing write tid 1 to smithi16530038-16 2024-09-16T04:42:13.267 INFO:tasks.rados.rados.0.smithi165.stdout:191: finishing copy_from racing read to smithi16530038-15 2024-09-16T04:42:13.268 INFO:tasks.rados.rados.0.smithi165.stdout:196: finishing write tid 2 to smithi16530038-16 2024-09-16T04:42:13.268 INFO:tasks.rados.rados.0.smithi165.stdout:192: finishing rollback tid 1 to smithi16530038-19 2024-09-16T04:42:13.268 INFO:tasks.rados.rados.0.smithi165.stdout:193: finishing copy_from racing read to smithi16530038-21 2024-09-16T04:42:13.268 INFO:tasks.rados.rados.0.smithi165.stdout:195: expect (ObjNum 75 snap 6 seq_num 75) 2024-09-16T04:42:13.298 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:13.295+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:13.322 INFO:tasks.rados.rados.0.smithi165.stdout:196: finishing write tid 3 to smithi16530038-16 2024-09-16T04:42:13.322 INFO:tasks.rados.rados.0.smithi165.stdout:196: finishing write tid 4 to smithi16530038-16 2024-09-16T04:42:13.322 INFO:tasks.rados.rados.0.smithi165.stdout:194: finishing copy_from racing read to smithi16530038-3 2024-09-16T04:42:13.322 INFO:tasks.rados.rados.0.smithi165.stdout:192: finishing rollback tid 2 to smithi16530038-19 2024-09-16T04:42:13.322 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 16 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:190: finishing write tid 1 to smithi16530038-13 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:192: done (8 left) 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:195: done (7 left) 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:200: done (6 left) 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:201: snap_create 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:190: finishing write tid 2 to smithi16530038-13 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:190: finishing write tid 3 to smithi16530038-13 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:191: finishing copy_from to smithi16530038-15 2024-09-16T04:42:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:196: finishing write tid 5 to smithi16530038-16 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:196: oid 16 updating version 0 to 42 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:196: oid 16 version 42 is already newer than 40 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:196: oid 16 version 42 is already newer than 39 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:196: oid 16 version 42 is already newer than 41 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 42 (ObjNum 87 snap 14 seq_num 87) dirty exists 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:196: left oid 16 (ObjNum 87 snap 14 seq_num 87) 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:193: finishing copy_from to smithi16530038-21 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:190: finishing write tid 4 to smithi16530038-13 2024-09-16T04:42:13.324 INFO:tasks.rados.rados.0.smithi165.stdout:194: finishing copy_from to smithi16530038-3 2024-09-16T04:42:13.325 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:13.334 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:13.334 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:13.356 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:13.357 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:13.361 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:13.361 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:13.377 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:42:13.380 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:13.388 INFO:tasks.rados.rados.0.smithi165.stdout:190: finishing write tid 5 to smithi16530038-13 2024-09-16T04:42:13.388 INFO:tasks.rados.rados.0.smithi165.stdout:190: oid 13 updating version 0 to 29 2024-09-16T04:42:13.388 INFO:tasks.rados.rados.0.smithi165.stdout:190: oid 13 updating version 29 to 32 2024-09-16T04:42:13.388 INFO:tasks.rados.rados.0.smithi165.stdout:190: oid 13 version 32 is already newer than 31 2024-09-16T04:42:13.388 INFO:tasks.rados.rados.0.smithi165.stdout:190: oid 13 version 32 is already newer than 30 2024-09-16T04:42:13.388 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 32 (ObjNum 86 snap 14 seq_num 86) dirty exists 2024-09-16T04:42:13.388 INFO:tasks.rados.rados.0.smithi165.stdout:190: left oid 13 (ObjNum 86 snap 14 seq_num 86) 2024-09-16T04:42:13.399 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:13.395+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:13.492 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:13.493 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:13.499 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:13.495+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:13.501 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:13.501 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:13.515 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:42:13.516 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:13.516 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:13.537 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:13.537 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:13.555 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:13.555 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:13.555 INFO:teuthology.orchestra.run.smithi059.stdout: "epoch": 63, 2024-09-16T04:42:13.555 INFO:teuthology.orchestra.run.smithi059.stdout: "fsid": "bdbec34e-c8aa-4776-9d32-ea336a99655f", 2024-09-16T04:42:13.555 INFO:teuthology.orchestra.run.smithi059.stdout: "created": "2024-09-16T04:41:10.374765+0000", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "modified": "2024-09-16T04:42:13.250299+0000", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "last_up_change": "2024-09-16T04:41:24.359002+0000", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "last_in_change": "2024-09-16T04:41:15.004474+0000", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "flags_num": 5799936, 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "flags_set": [ 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "pglog_hardlimit", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "purged_snapdirs", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "recovery_deletes", 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "sortbitwise" 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:13.556 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_version": 6, 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_snapshot": "", 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "pool_max": 3, 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "max_osd": 16, 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "require_min_compat_client": "luminous", 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "min_compat_client": "jewel", 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "require_osd_release": "squid", 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "allow_crimson": false, 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: "pools": [ 2024-09-16T04:42:13.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 1, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "pool_name": ".mgr", 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "create_time": "2024-09-16T04:41:23.074365+0000", 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "flags": 1, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "flags_names": "hashpspool", 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "type": 1, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 2, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "min_size": 1, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_rule": 0, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:42:13.558 INFO:teuthology.orchestra.run.smithi059.stdout: "is_stretch_pool": false, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "object_hash": 2, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num": 1, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_placement_num": 1, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_placement_num_target": 1, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_target": 1, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_pending": 1, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "last_pg_merge_meta": { 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "source_pgid": "0.0", 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "ready_epoch": 0, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "last_epoch_started": 0, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "last_epoch_clean": 0, 2024-09-16T04:42:13.559 INFO:teuthology.orchestra.run.smithi059.stdout: "source_version": "0'0", 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "target_version": "0'0" 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "last_change": "25", 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend": "0", 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "auid": 0, 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_seq": 0, 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_epoch": 0, 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "pool_snaps": [], 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "removed_snaps": "[]", 2024-09-16T04:42:13.560 INFO:teuthology.orchestra.run.smithi059.stdout: "quota_max_bytes": 0, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "quota_max_objects": 0, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "tiers": [], 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "tier_of": -1, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "read_tier": -1, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "write_tier": -1, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_mode": "none", 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_bytes": 0, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_objects": 0, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_min_flush_age": 0, 2024-09-16T04:42:13.561 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_min_evict_age": 0, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "erasure_code_profile": "", 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_params": { 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "none" 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_period": 0, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_count": 0, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "use_gmt_hitset": true, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "grade_table": [], 2024-09-16T04:42:13.562 INFO:teuthology.orchestra.run.smithi059.stdout: "stripe_width": 0, 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "expected_num_objects": 0, 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "fast_read": false, 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "options": { 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_max": 32, 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_min": 1 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "application_metadata": { 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": {} 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "read_balance": { 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "score_type": "Fair distribution", 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "score_acting": 15.380000114440918, 2024-09-16T04:42:13.563 INFO:teuthology.orchestra.run.smithi059.stdout: "score_stable": 15.380000114440918, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "optimal_score": 0.12999999523162842, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "raw_score_acting": 2, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "raw_score_stable": 2, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity_weighted": 1, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "average_primary_affinity": 1, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "average_primary_affinity_weighted": 1 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 2, 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "pool_name": "rbd", 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "create_time": "2024-09-16T04:41:25.522114+0000", 2024-09-16T04:42:13.564 INFO:teuthology.orchestra.run.smithi059.stdout: "flags": 8193, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "type": 1, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 2, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "min_size": 1, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_rule": 0, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "is_stretch_pool": false, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "object_hash": 2, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num": 8, 2024-09-16T04:42:13.565 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_placement_num": 8, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_placement_num_target": 8, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_target": 8, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_pending": 8, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "last_pg_merge_meta": { 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "source_pgid": "0.0", 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "ready_epoch": 0, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "last_epoch_started": 0, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "last_epoch_clean": 0, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "source_version": "0'0", 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "target_version": "0'0" 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.566 INFO:teuthology.orchestra.run.smithi059.stdout: "last_change": "28", 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend": "0", 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "auid": 0, 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_seq": 2, 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_epoch": 28, 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "pool_snaps": [], 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "removed_snaps": "[]", 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "quota_max_bytes": 0, 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "quota_max_objects": 0, 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "tiers": [], 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "tier_of": -1, 2024-09-16T04:42:13.567 INFO:teuthology.orchestra.run.smithi059.stdout: "read_tier": -1, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "write_tier": -1, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_mode": "none", 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_bytes": 0, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_objects": 0, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_min_flush_age": 0, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_min_evict_age": 0, 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "erasure_code_profile": "", 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_params": { 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "none" 2024-09-16T04:42:13.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_period": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_count": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "use_gmt_hitset": true, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "grade_table": [], 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "stripe_width": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "expected_num_objects": 0, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "fast_read": false, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "options": {}, 2024-09-16T04:42:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: "application_metadata": { 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd": {} 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "read_balance": { 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "score_type": "Fair distribution", 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "score_acting": 4.0199999809265137, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "score_stable": 4.0199999809265137, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "optimal_score": 0.56000000238418579, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "raw_score_acting": 2.25, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "raw_score_stable": 2.25, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity_weighted": 1, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "average_primary_affinity": 1, 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: "average_primary_affinity_weighted": 1 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 3, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "pool_name": "unique_pool_0", 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "create_time": "2024-09-16T04:41:39.714918+0000", 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "flags": 8193, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "type": 3, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 7, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "min_size": 5, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_rule": 1, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:42:13.571 INFO:teuthology.orchestra.run.smithi059.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "is_stretch_pool": false, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "object_hash": 2, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num": 26, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_placement_num": 26, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_placement_num_target": 36, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_target": 36, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_num_pending": 26, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "last_pg_merge_meta": { 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "source_pgid": "3.18", 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "ready_epoch": 51, 2024-09-16T04:42:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: "last_epoch_started": 50, 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "last_epoch_clean": 50, 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "source_version": "35'2", 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "target_version": "44'8" 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "last_change": "63", 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend": "0", 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend_prenautilus": "51", 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "last_force_op_resend_preluminous": "55", 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "auid": 0, 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_seq": 15, 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "snap_epoch": 63, 2024-09-16T04:42:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: "pool_snaps": [], 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "removed_snaps": "[]", 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "quota_max_bytes": 0, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "quota_max_objects": 0, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "tiers": [], 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "tier_of": -1, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "read_tier": -1, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "write_tier": -1, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_mode": "none", 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_bytes": 0, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_objects": 0, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:42:13.574 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_min_flush_age": 0, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "cache_min_evict_age": 0, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "erasure_code_profile": "shecprofile", 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_params": { 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "none" 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_period": 0, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_count": 0, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "use_gmt_hitset": true, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:42:13.575 INFO:teuthology.orchestra.run.smithi059.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "grade_table": [], 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "stripe_width": 16384, 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "expected_num_objects": 0, 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "fast_read": false, 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "options": {}, 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "application_metadata": { 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "rados": {} 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "osds": [ 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.576 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 0, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "b9529dc0-e1eb-4b2f-8297-45254ddbffa8", 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 21, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 41, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.577 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6800", 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6801", 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.578 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6802", 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6803", 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6806", 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6807", 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.580 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6804", 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6805", 2024-09-16T04:42:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1228226348 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.59:6801/1228226348", 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.59:6803/1228226348", 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.59:6807/1228226348", 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.59:6805/1228226348", 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 1, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "dceca258-f5a1-4e8d-be6e-9d38bc5f14e1", 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 23, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 0, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6816", 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6817", 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6818", 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6819", 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.585 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6822", 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6823", 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6820", 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6821", 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2731115256 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.587 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.139:6817/2731115256", 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.139:6819/2731115256", 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.139:6823/2731115256", 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.139:6821/2731115256", 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 2, 2024-09-16T04:42:13.588 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "6bfc934a-0ca2-4b1f-9e09-14c18a7d5116", 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 32, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6824", 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6825", 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6826", 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6827", 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6830", 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6831", 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6828", 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6829", 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 76637888 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.593 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.144:6825/76637888", 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.144:6827/76637888", 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.144:6831/76637888", 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.144:6829/76637888", 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 3, 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "0b949ef9-82e3-483f-8be6-65e04d169c4e", 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.594 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 53, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.595 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6802", 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6803", 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.596 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6804", 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6805", 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6808", 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6809", 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.598 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6806", 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6807", 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 58455506 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.165:6803/58455506", 2024-09-16T04:42:13.599 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.165:6805/58455506", 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.165:6809/58455506", 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.165:6807/58455506", 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 4, 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "d35e5791-7bb4-493d-99ec-3ff942117371", 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.600 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 41, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6824", 2024-09-16T04:42:13.601 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6825", 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.602 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6826", 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6827", 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6830", 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6831", 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.604 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6828", 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6829", 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2712399685 2024-09-16T04:42:13.605 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.59:6825/2712399685", 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.59:6827/2712399685", 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.59:6831/2712399685", 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.59:6829/2712399685", 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.606 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 5, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "159725f2-3713-4f0e-9e44-55b678bf09fa", 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 60, 2024-09-16T04:42:13.607 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6800", 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6801", 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.608 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6802", 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6803", 2024-09-16T04:42:13.609 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6806", 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6807", 2024-09-16T04:42:13.610 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6804", 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.611 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6805", 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2946170755 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.139:6801/2946170755", 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.139:6803/2946170755", 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.139:6807/2946170755", 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.139:6805/2946170755", 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 6, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "e642cfb3-cf94-4124-93db-9e2bdf1b7c5f", 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 25, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.613 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6800", 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6801", 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.614 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6802", 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6803", 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6806", 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6807", 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6804", 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6805", 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3812816248 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.617 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.144:6801/3812816248", 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.144:6803/3812816248", 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.144:6807/3812816248", 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.144:6805/3812816248", 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 7, 2024-09-16T04:42:13.618 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb", 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 32, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6818", 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6819", 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.620 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6820", 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6821", 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.621 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6824", 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6825", 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6822", 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6823", 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1352862134 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.165:6819/1352862134", 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.165:6821/1352862134", 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.165:6825/1352862134", 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.165:6823/1352862134", 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 8, 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "04681c69-387b-4dd8-9eb4-687f34ba9dab", 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.624 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 57, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6816", 2024-09-16T04:42:13.625 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6817", 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.626 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6818", 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6819", 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.627 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6822", 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6823", 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6820", 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6821", 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1451673997 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.59:6817/1451673997", 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.59:6819/1451673997", 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.59:6823/1451673997", 2024-09-16T04:42:13.629 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.59:6821/1451673997", 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 9, 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "713a3b76-8daf-4e44-9d0b-68369a9da692", 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.630 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 23, 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 54, 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6808", 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6809", 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.632 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6810", 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6811", 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.633 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6814", 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6815", 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.634 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6812", 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6813", 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 174106571 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.139:6809/174106571", 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.139:6811/174106571", 2024-09-16T04:42:13.635 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.139:6815/174106571", 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.139:6813/174106571", 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 10, 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "81450492-3178-4643-8066-847fcdd266b4", 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.636 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 53, 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6808", 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6809", 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6810", 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6812", 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6817", 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6818", 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.640 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6813", 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6815", 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 66779937 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.144:6809/66779937", 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.144:6812/66779937", 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.144:6818/66779937", 2024-09-16T04:42:13.641 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.144:6815/66779937", 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 11, 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "9938b272-43e2-4b48-91ca-42cbe5f26664", 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.642 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 21, 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 53, 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6810", 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.643 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6811", 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6812", 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6813", 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6816", 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6817", 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6814", 2024-09-16T04:42:13.646 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6815", 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 49204916 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.165:6811/49204916", 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.165:6813/49204916", 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.165:6817/49204916", 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.165:6815/49204916", 2024-09-16T04:42:13.647 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 12, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "cd20410e-c17f-43d7-8512-9efd8193a251", 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.648 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 0, 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6808", 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.649 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6809", 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6810", 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.650 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6811", 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6814", 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.651 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6815", 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6812", 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.652 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:6813", 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3097346146 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.59:6809/3097346146", 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.59:6811/3097346146", 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.59:6815/3097346146", 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.59:6813/3097346146", 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 13, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "953622d2-9e72-451d-8863-d8c55139c65b", 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 23, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 61, 2024-09-16T04:42:13.654 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6824", 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6825", 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.655 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6826", 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6827", 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.656 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6830", 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6831", 2024-09-16T04:42:13.657 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6828", 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.139:6829", 2024-09-16T04:42:13.658 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2903817823 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.139:6825/2903817823", 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.139:6827/2903817823", 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.139:6831/2903817823", 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.139:6829/2903817823", 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.659 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 14, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "e6388924-0444-47c0-b61c-7ca66661237f", 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 58, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.660 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6811", 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6814", 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.661 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6816", 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6819", 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.662 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6822", 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6823", 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.663 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6820", 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.144:6821", 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 229023981 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.664 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.144:6814/229023981", 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.144:6819/229023981", 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.144:6823/229023981", 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.144:6821/229023981", 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 15, 2024-09-16T04:42:13.665 INFO:teuthology.orchestra.run.smithi059.stdout: "uuid": "8335941d-9563-4527-90f6-3b7f9129f530", 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "up": 1, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "in": 1, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "weight": 1, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_affinity": 1, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_begin": 0, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "last_clean_end": 0, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "up_from": 22, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "up_thru": 53, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "down_at": 0, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "lost_at": 0, 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrs": { 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.666 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6826", 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6827", 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addrs": { 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.667 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6828", 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6829", 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addrs": { 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.668 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6832", 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6833", 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addrs": { 2024-09-16T04:42:13.669 INFO:teuthology.orchestra.run.smithi059.stdout: "addrvec": [ 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v2", 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6830", 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.165:6831", 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 2758676961 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.670 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "172.21.15.165:6827/2758676961", 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "172.21.15.165:6829/2758676961", 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_back_addr": "172.21.15.165:6833/2758676961", 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_front_addr": "172.21.15.165:6831/2758676961", 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "state": [ 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "exists", 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "up" 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_xinfo": [ 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 0, 2024-09-16T04:42:13.671 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.223183+0000", 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 1, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.672 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.481924+0000", 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 2, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.389820+0000", 2024-09-16T04:42:13.673 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 3, 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.279164+0000", 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.674 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 4, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.793764+0000", 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 5, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.675 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.571561+0000", 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 6, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.265206+0000", 2024-09-16T04:42:13.676 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 7, 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.684523+0000", 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.677 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 8, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.452789+0000", 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 9, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.678 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.332716+0000", 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 10, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.258982+0000", 2024-09-16T04:42:13.679 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 11, 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.166555+0000", 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 12, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.256846+0000", 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 13, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.681 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.572294+0000", 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 14, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.852615+0000", 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.682 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": 15, 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "down_stamp": "0.000000", 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_probability": 0, 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "laggy_interval": 0, 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "features": 4540701547738038271, 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "old_weight": 0, 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "last_purged_snaps_scrub": "2024-09-16T04:41:21.867981+0000", 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "dead_epoch": 0 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_upmap": [], 2024-09-16T04:42:13.683 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_upmap_items": [], 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_upmap_primaries": [], 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: "pg_temp": [ 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: "pgid": "3.19", 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osds": [ 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: 14, 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: 1, 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: 0, 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: 3, 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: 15, 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: 8, 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: 10 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "primary_temp": [], 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklist": {}, 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "range_blocklist": {}, 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "erasure_code_profiles": { 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "default": { 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "crush-failure-domain": "osd", 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "k": "2", 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "m": "1", 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin": "jerasure", 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "technique": "reed_sol_van" 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "shecprofile": { 2024-09-16T04:42:13.685 INFO:teuthology.orchestra.run.smithi059.stdout: "c": "2", 2024-09-16T04:42:13.688 INFO:teuthology.orchestra.run.smithi059.stdout: "crush-device-class": "", 2024-09-16T04:42:13.688 INFO:teuthology.orchestra.run.smithi059.stdout: "crush-failure-domain": "osd", 2024-09-16T04:42:13.688 INFO:teuthology.orchestra.run.smithi059.stdout: "crush-num-failure-domains": "0", 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "crush-osds-per-failure-domain": "0", 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "crush-root": "default", 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "k": "4", 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "m": "3", 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "shecprofile", 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin": "shec", 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "technique": "multiple" 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "removed_snaps_queue": [ 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 3, 2024-09-16T04:42:13.689 INFO:teuthology.orchestra.run.smithi059.stdout: "snaps": [ 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: "begin": 2, 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: "length": 5 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: "begin": 8, 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: "length": 2 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: "begin": 12, 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: "length": 1 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.690 INFO:teuthology.orchestra.run.smithi059.stdout: "begin": 14, 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: "length": 1 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: "new_removed_snaps": [ 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 3, 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: "snaps": [ 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: "begin": 14, 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: "length": 1 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "new_purged_snaps": [], 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_node_flags": {}, 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "device_class_flags": {}, 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "stretch_mode": { 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "stretch_mode_enabled": false, 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "stretch_bucket_count": 0, 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "degraded_stretch_mode": 0, 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "recovering_stretch_mode": 0, 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: "stretch_mode_bucket": 0 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:13.692 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:13.694 INFO:tasks.rados.rados.0.smithi165.stdout:199: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:13.694 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:13.695 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:13.695 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:13.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.695 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 63/32 ECSubWriteReply(tid=54, last_complete=63'34, committed=1, applied=1))", 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:13.593180+0000", 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.058538673999999999, 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058579566999999999, 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593180+0000", 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593180+0000", 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593181+0000", 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2249999999999999e-06 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593193+0000", 2024-09-16T04:42:13.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1768e-05 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593210+0000", 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7030000000000001e-05 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593223+0000", 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3220999999999999e-05 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 63/32 ECSubWriteReply(tid=54, last_complete=63'34, committed=1, applied=1))", 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:13.593336+0000", 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.058382360000000001, 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058511002999999999, 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593336+0000", 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593336+0000", 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593337+0000", 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1800000000000005e-07 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:13.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593340+0000", 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.659e-06 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593351+0000", 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0922e-05 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593357+0000", 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9499999999999998e-06 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:13.701 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 63/32 ECSubWriteReply(tid=54, last_complete=63'34, committed=1, applied=1))", 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:13.593568+0000", 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.058150110999999997, 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058364448999999999, 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:13.702 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593568+0000", 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593568+0000", 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:13.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593569+0000", 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7100000000000001e-07 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593573+0000", 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9940000000000002e-06 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593585+0000", 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1647e-05 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593590+0000", 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.677e-06 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 63/32 ECSubWriteReply(tid=54, last_complete=63'34, committed=1, applied=1))", 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:13.593753+0000", 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.057965029000000001, 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058326332000000002, 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:13.705 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593753+0000", 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593753+0000", 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593754+0000", 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7700000000000004e-07 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593758+0000", 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593768+0000", 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7620000000000007e-06 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593772+0000", 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.826e-06 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.708 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 63/32 ECSubWriteReply(tid=54, last_complete=63'34, committed=1, applied=1))", 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:13.593854+0000", 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.057864744000000003, 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058375165999999999, 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593854+0000", 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593854+0000", 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593854+0000", 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.37e-07 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593858+0000", 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3299999999999999e-06 2024-09-16T04:42:13.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593867+0000", 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1710000000000008e-06 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:13.593871+0000", 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6959999999999996e-06 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:13.711 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:13.712 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 5 2024-09-16T04:42:13.712 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:13.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:42:13.713 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:13.692+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:13.714 INFO:tasks.thrashosds.thrasher:j is {'epoch': 63, 'fsid': 'bdbec34e-c8aa-4776-9d32-ea336a99655f', 'created': '2024-09-16T04:41:10.374765+0000', 'modified': '2024-09-16T04:42:13.250299+0000', 'last_up_change': '2024-09-16T04:41:24.359002+0000', 'last_in_change': '2024-09-16T04:41:15.004474+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T04:41:23.074365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T04:41:25.522114+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.019999980926514, 'score_stable': 4.019999980926514, 'optimal_score': 0.5600000023841858, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T04:41:39.714918+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '3.18', 'ready_epoch': 51, 'last_epoch_started': 50, 'last_epoch_clean': 50, 'source_version': "35'2", 'target_version': "44'8"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '51', 'last_force_op_resend_preluminous': '55', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'b9529dc0-e1eb-4b2f-8297-45254ddbffa8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6800', 'nonce': 1228226348}, {'type': 'v1', 'addr': '172.21.15.59:6801', 'nonce': 1228226348}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6802', 'nonce': 1228226348}, {'type': 'v1', 'addr': '172.21.15.59:6803', 'nonce': 1228226348}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6806', 'nonce': 1228226348}, {'type': 'v1', 'addr': '172.21.15.59:6807', 'nonce': 1228226348}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6804', 'nonce': 1228226348}, {'type': 'v1', 'addr': '172.21.15.59:6805', 'nonce': 1228226348}]}, 'public_addr': '172.21.15.59:6801/1228226348', 'cluster_addr': '172.21.15.59:6803/1228226348', 'heartbeat_back_addr': '172.21.15.59:6807/1228226348', 'heartbeat_front_addr': '172.21.15.59:6805/1228226348', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'dceca258-f5a1-4e8d-be6e-9d38bc5f14e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6816', 'nonce': 2731115256}, {'type': 'v1', 'addr': '172.21.15.139:6817', 'nonce': 2731115256}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6818', 'nonce': 2731115256}, {'type': 'v1', 'addr': '172.21.15.139:6819', 'nonce': 2731115256}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6822', 'nonce': 2731115256}, {'type': 'v1', 'addr': '172.21.15.139:6823', 'nonce': 2731115256}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6820', 'nonce': 2731115256}, {'type': 'v1', 'addr': '172.21.15.139:6821', 'nonce': 2731115256}]}, 'public_addr': '172.21.15.139:6817/2731115256', 'cluster_addr': '172.21.15.139:6819/2731115256', 'heartbeat_back_addr': '172.21.15.139:6823/2731115256', 'heartbeat_front_addr': '172.21.15.139:6821/2731115256', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6bfc934a-0ca2-4b1f-9e09-14c18a7d5116', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6824', 'nonce': 76637888}, {'type': 'v1', 'addr': '172.21.15.144:6825', 'nonce': 76637888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6826', 'nonce': 76637888}, {'type': 'v1', 'addr': '172.21.15.144:6827', 'nonce': 76637888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6830', 'nonce': 76637888}, {'type': 'v1', 'addr': '172.21.15.144:6831', 'nonce': 76637888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6828', 'nonce': 76637888}, {'type': 'v1', 'addr': '172.21.15.144:6829', 'nonce': 76637888}]}, 'public_addr': '172.21.15.144:6825/76637888', 'cluster_addr': '172.21.15.144:6827/76637888', 'heartbeat_back_addr': '172.21.15.144:6831/76637888', 'heartbeat_front_addr': '172.21.15.144:6829/76637888', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0b949ef9-82e3-483f-8be6-65e04d169c4e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6802', 'nonce': 58455506}, {'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 58455506}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6804', 'nonce': 58455506}, {'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 58455506}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6808', 'nonce': 58455506}, {'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 58455506}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6806', 'nonce': 58455506}, {'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 58455506}]}, 'public_addr': '172.21.15.165:6803/58455506', 'cluster_addr': '172.21.15.165:6805/58455506', 'heartbeat_back_addr': '172.21.15.165:6809/58455506', 'heartbeat_front_addr': '172.21.15.165:6807/58455506', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd35e5791-7bb4-493d-99ec-3ff942117371', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6824', 'nonce': 2712399685}, {'type': 'v1', 'addr': '172.21.15.59:6825', 'nonce': 2712399685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6826', 'nonce': 2712399685}, {'type': 'v1', 'addr': '172.21.15.59:6827', 'nonce': 2712399685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6830', 'nonce': 2712399685}, {'type': 'v1', 'addr': '172.21.15.59:6831', 'nonce': 2712399685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6828', 'nonce': 2712399685}, {'type': 'v1', 'addr': '172.21.15.59:6829', 'nonce': 2712399685}]}, 'public_addr': '172.21.15.59:6825/2712399685', 'cluster_addr': '172.21.15.59:6827/2712399685', 'heartbeat_back_addr': '172.21.15.59:6831/2712399685', 'heartbeat_front_addr': '172.21.15.59:6829/2712399685', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '159725f2-3713-4f0e-9e44-55b678bf09fa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 60, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6800', 'nonce': 2946170755}, {'type': 'v1', 'addr': '172.21.15.139:6801', 'nonce': 2946170755}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6802', 'nonce': 2946170755}, {'type': 'v1', 'addr': '172.21.15.139:6803', 'nonce': 2946170755}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6806', 'nonce': 2946170755}, {'type': 'v1', 'addr': '172.21.15.139:6807', 'nonce': 2946170755}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6804', 'nonce': 2946170755}, {'type': 'v1', 'addr': '172.21.15.139:6805', 'nonce': 2946170755}]}, 'public_addr': '172.21.15.139:6801/2946170755', 'cluster_addr': '172.21.15.139:6803/2946170755', 'heartbeat_back_addr': '172.21.15.139:6807/2946170755', 'heartbeat_front_addr': '172.21.15.139:6805/2946170755', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e642cfb3-cf94-4124-93db-9e2bdf1b7c5f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6800', 'nonce': 3812816248}, {'type': 'v1', 'addr': '172.21.15.144:6801', 'nonce': 3812816248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6802', 'nonce': 3812816248}, {'type': 'v1', 'addr': '172.21.15.144:6803', 'nonce': 3812816248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6806', 'nonce': 3812816248}, {'type': 'v1', 'addr': '172.21.15.144:6807', 'nonce': 3812816248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6804', 'nonce': 3812816248}, {'type': 'v1', 'addr': '172.21.15.144:6805', 'nonce': 3812816248}]}, 'public_addr': '172.21.15.144:6801/3812816248', 'cluster_addr': '172.21.15.144:6803/3812816248', 'heartbeat_back_addr': '172.21.15.144:6807/3812816248', 'heartbeat_front_addr': '172.21.15.144:6805/3812816248', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6818', 'nonce': 1352862134}, {'type': 'v1', 'addr': '172.21.15.165:6819', 'nonce': 1352862134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6820', 'nonce': 1352862134}, {'type': 'v1', 'addr': '172.21.15.165:6821', 'nonce': 1352862134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6824', 'nonce': 1352862134}, {'type': 'v1', 'addr': '172.21.15.165:6825', 'nonce': 1352862134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6822', 'nonce': 1352862134}, {'type': 'v1', 'addr': '172.21.15.165:6823', 'nonce': 1352862134}]}, 'public_addr': '172.21.15.165:6819/1352862134', 'cluster_addr': '172.21.15.165:6821/1352862134', 'heartbeat_back_addr': '172.21.15.165:6825/1352862134', 'heartbeat_front_addr': '172.21.15.165:6823/1352862134', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '04681c69-387b-4dd8-9eb4-687f34ba9dab', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 57, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6816', 'nonce': 1451673997}, {'type': 'v1', 'addr': '172.21.15.59:6817', 'nonce': 1451673997}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6818', 'nonce': 1451673997}, {'type': 'v1', 'addr': '172.21.15.59:6819', 'nonce': 1451673997}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6822', 'nonce': 1451673997}, {'type': 'v1', 'addr': '172.21.15.59:6823', 'nonce': 1451673997}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6820', 'nonce': 1451673997}, {'type': 'v1', 'addr': '172.21.15.59:6821', 'nonce': 1451673997}]}, 'public_addr': '172.21.15.59:6817/1451673997', 'cluster_addr': '172.21.15.59:6819/1451673997', 'heartbeat_back_addr': '172.21.15.59:6823/1451673997', 'heartbeat_front_addr': '172.21.15.59:6821/1451673997', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '713a3b76-8daf-4e44-9d0b-68369a9da692', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6808', 'nonce': 174106571}, {'type': 'v1', 'addr': '172.21.15.139:6809', 'nonce': 174106571}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6810', 'nonce': 174106571}, {'type': 'v1', 'addr': '172.21.15.139:6811', 'nonce': 174106571}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6814', 'nonce': 174106571}, {'type': 'v1', 'addr': '172.21.15.139:6815', 'nonce': 174106571}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6812', 'nonce': 174106571}, {'type': 'v1', 'addr': '172.21.15.139:6813', 'nonce': 174106571}]}, 'public_addr': '172.21.15.139:6809/174106571', 'cluster_addr': '172.21.15.139:6811/174106571', 'heartbeat_back_addr': '172.21.15.139:6815/174106571', 'heartbeat_front_addr': '172.21.15.139:6813/174106571', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '81450492-3178-4643-8066-847fcdd266b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6808', 'nonce': 66779937}, {'type': 'v1', 'addr': '172.21.15.144:6809', 'nonce': 66779937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6810', 'nonce': 66779937}, {'type': 'v1', 'addr': '172.21.15.144:6812', 'nonce': 66779937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6817', 'nonce': 66779937}, {'type': 'v1', 'addr': '172.21.15.144:6818', 'nonce': 66779937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6813', 'nonce': 66779937}, {'type': 'v1', 'addr': '172.21.15.144:6815', 'nonce': 66779937}]}, 'public_addr': '172.21.15.144:6809/66779937', 'cluster_addr': '172.21.15.144:6812/66779937', 'heartbeat_back_addr': '172.21.15.144:6818/66779937', 'heartbeat_front_addr': '172.21.15.144:6815/66779937', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '9938b272-43e2-4b48-91ca-42cbe5f26664', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6810', 'nonce': 49204916}, {'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 49204916}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6812', 'nonce': 49204916}, {'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 49204916}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6816', 'nonce': 49204916}, {'type': 'v1', 'addr': '172.21.15.165:6817', 'nonce': 49204916}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6814', 'nonce': 49204916}, {'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 49204916}]}, 'public_addr': '172.21.15.165:6811/49204916', 'cluster_addr': '172.21.15.165:6813/49204916', 'heartbeat_back_addr': '172.21.15.165:6817/49204916', 'heartbeat_front_addr': '172.21.15.165:6815/49204916', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'cd20410e-c17f-43d7-8512-9efd8193a251', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6808', 'nonce': 3097346146}, {'type': 'v1', 'addr': '172.21.15.59:6809', 'nonce': 3097346146}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6810', 'nonce': 3097346146}, {'type': 'v1', 'addr': '172.21.15.59:6811', 'nonce': 3097346146}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6814', 'nonce': 3097346146}, {'type': 'v1', 'addr': '172.21.15.59:6815', 'nonce': 3097346146}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6812', 'nonce': 3097346146}, {'type': 'v1', 'addr': '172.21.15.59:6813', 'nonce': 3097346146}]}, 'public_addr': '172.21.15.59:6809/3097346146', 'cluster_addr': '172.21.15.59:6811/3097346146', 'heartbeat_back_addr': '172.21.15.59:6815/3097346146', 'heartbeat_front_addr': '172.21.15.59:6813/3097346146', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '953622d2-9e72-451d-8863-d8c55139c65b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6824', 'nonce': 2903817823}, {'type': 'v1', 'addr': '172.21.15.139:6825', 'nonce': 2903817823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 2903817823}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 2903817823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6830', 'nonce': 2903817823}, {'type': 'v1', 'addr': '172.21.15.139:6831', 'nonce': 2903817823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2903817823}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2903817823}]}, 'public_addr': '172.21.15.139:6825/2903817823', 'cluster_addr': '172.21.15.139:6827/2903817823', 'heartbeat_back_addr': '172.21.15.139:6831/2903817823', 'heartbeat_front_addr': '172.21.15.139:6829/2903817823', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': 'e6388924-0444-47c0-b61c-7ca66661237f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6811', 'nonce': 229023981}, {'type': 'v1', 'addr': '172.21.15.144:6814', 'nonce': 229023981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6816', 'nonce': 229023981}, {'type': 'v1', 'addr': '172.21.15.144:6819', 'nonce': 229023981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6822', 'nonce': 229023981}, {'type': 'v1', 'addr': '172.21.15.144:6823', 'nonce': 229023981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6820', 'nonce': 229023981}, {'type': 'v1', 'addr': '172.21.15.144:6821', 'nonce': 229023981}]}, 'public_addr': '172.21.15.144:6814/229023981', 'cluster_addr': '172.21.15.144:6819/229023981', 'heartbeat_back_addr': '172.21.15.144:6823/229023981', 'heartbeat_front_addr': '172.21.15.144:6821/229023981', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '8335941d-9563-4527-90f6-3b7f9129f530', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6826', 'nonce': 2758676961}, {'type': 'v1', 'addr': '172.21.15.165:6827', 'nonce': 2758676961}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6828', 'nonce': 2758676961}, {'type': 'v1', 'addr': '172.21.15.165:6829', 'nonce': 2758676961}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6832', 'nonce': 2758676961}, {'type': 'v1', 'addr': '172.21.15.165:6833', 'nonce': 2758676961}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6830', 'nonce': 2758676961}, {'type': 'v1', 'addr': '172.21.15.165:6831', 'nonce': 2758676961}]}, 'public_addr': '172.21.15.165:6827/2758676961', 'cluster_addr': '172.21.15.165:6829/2758676961', 'heartbeat_back_addr': '172.21.15.165:6833/2758676961', 'heartbeat_front_addr': '172.21.15.165:6831/2758676961', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.223183+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.481924+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.389820+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.279164+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.793764+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.571561+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.265206+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.684523+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.452789+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.332716+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.258982+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.166555+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.256846+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.572294+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.852615+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:41:21.867981+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.19', 'osds': [14, 1, 0, 3, 15, 8, 10]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 5}, {'begin': 8, 'length': 2}, {'begin': 12, 'length': 1}, {'begin': 14, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T04:42:13.714 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:13.715 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:13.715 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:13.715 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:13.716 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:13.716 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:13.716 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:13.716 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:13.717 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:13.717 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:13.717 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:13.717 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:13.718 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:13.718 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:13.718 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:13.721 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:13.721 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:13.742 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:13.743 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:13.795 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:13.791+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:13.865 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:13.865 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:13.865 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:13.866 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:13.866 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:13.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:42:13.897 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:13.892+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:13.998 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:13.991+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:13.999 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:13.999 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:13.999 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:119 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'81, mlcod=28'78)", 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 40.552011147000002, 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20259546, 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.000 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444890+0000", 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.444891+0000", 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6360000000000001e-06 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.001 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.445043+0000", 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000151581 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.445078+0000", 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5077000000000003e-05 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.545224+0000", 2024-09-16T04:42:14.002 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10014579899999999 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.645477+0000", 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100253114 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.645521+0000", 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4218999999999998e-05 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.647416+0000", 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001895477 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:33.647485+0000", 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8557000000000001e-05 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:146 1.0 e29/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 29'104, mlcod=28'102)", 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:42:14.004 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.583196428999997, 2024-09-16T04:42:14.005 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20228226799999999, 2024-09-16T04:42:14.005 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.005 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.005 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.005 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.005 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.005 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.006 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:42:14.006 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.006 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.006 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.006 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.006 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413704+0000", 2024-09-16T04:42:14.007 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.007 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.007 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.007 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.007 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413706+0000", 2024-09-16T04:42:14.007 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7400000000000001e-06 2024-09-16T04:42:14.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.008 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.008 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.008 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413718+0000", 2024-09-16T04:42:14.008 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1892000000000001e-05 2024-09-16T04:42:14.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.413749+0000", 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0848e-05 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.513883+0000", 2024-09-16T04:42:14.009 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100134561 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.614137+0000", 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100254093 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.614197+0000", 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9703000000000001e-05 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.010 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.615867+0000", 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001670073 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:35.615987+0000", 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000119358 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:202 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'148, mlcod=29'145)", 2024-09-16T04:42:14.011 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 33.650629807999998, 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30312661499999999, 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.012 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.013 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346271+0000", 2024-09-16T04:42:14.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.013 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.013 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.013 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.013 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346273+0000", 2024-09-16T04:42:14.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7630000000000001e-06 2024-09-16T04:42:14.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.014 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.014 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.014 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346284+0000", 2024-09-16T04:42:14.014 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1489e-05 2024-09-16T04:42:14.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346311+0000", 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6888000000000001e-05 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346325+0000", 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3669000000000001e-05 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.015 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.016 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.445851+0000", 2024-09-16T04:42:14.016 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099526192999999999 2024-09-16T04:42:14.016 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.016 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.016 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.016 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.445885+0000", 2024-09-16T04:42:14.016 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4075000000000002e-05 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649211+0000", 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20332629399999999 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649397+0000", 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018624399999999999 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.017 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:203 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'149, mlcod=29'145)", 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 33.650161635000003, 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30271984099999999, 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346739+0000", 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.019 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346740+0000", 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0699999999999996e-07 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346874+0000", 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000134585 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.346885+0000", 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0283e-05 2024-09-16T04:42:14.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.021 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.021 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.021 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.446978+0000", 2024-09-16T04:42:14.021 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009309199999999 2024-09-16T04:42:14.021 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.021 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.022 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.022 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447153+0000", 2024-09-16T04:42:14.022 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000174721 2024-09-16T04:42:14.022 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.022 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.022 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.023 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447187+0000", 2024-09-16T04:42:14.023 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4913999999999998e-05 2024-09-16T04:42:14.023 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.023 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.023 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.023 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649408+0000", 2024-09-16T04:42:14.023 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20222030199999999 2024-09-16T04:42:14.024 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.024 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.024 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.024 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649459+0000", 2024-09-16T04:42:14.024 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1137000000000002e-05 2024-09-16T04:42:14.024 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.024 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:204 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'150, mlcod=29'145)", 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 33.549848322999999, 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246620800000001, 2024-09-16T04:42:14.025 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.026 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.026 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.026 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.026 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.026 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.026 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:42:14.026 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.027 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.027 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.027 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.027 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447052+0000", 2024-09-16T04:42:14.027 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.027 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.028 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.028 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.028 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447054+0000", 2024-09-16T04:42:14.028 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3430000000000001e-06 2024-09-16T04:42:14.028 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.028 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.028 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447066+0000", 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2055999999999999e-05 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447085+0000", 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9165e-05 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.029 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447099+0000", 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3791999999999999e-05 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448012+0000", 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000913319 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448042+0000", 2024-09-16T04:42:14.030 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0074e-05 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649463+0000", 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20142125599999999 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649519+0000", 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5203e-05 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.031 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:205 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'151, mlcod=29'145)", 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 33.549783861000002, 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202457419, 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.032 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447117+0000", 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447123+0000", 2024-09-16T04:42:14.033 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5600000000000001e-06 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447133+0000", 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7249999999999992e-06 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447140+0000", 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.204e-06 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.034 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448504+0000", 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013644340000000001 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448533+0000", 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8691e-05 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649523+0000", 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20098977900000001 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.035 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649574+0000", 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1523000000000001e-05 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:206 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'152, mlcod=29'145)", 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 33.549750760000002, 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20247812800000001, 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.036 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.037 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447150+0000", 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.03e-07 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447206+0000", 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5952999999999998e-05 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.038 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447217+0000", 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0533000000000001e-05 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.447228+0000", 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1012000000000001e-05 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448968+0000", 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001740412 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.039 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.448996+0000", 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7331999999999999e-05 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649579+0000", 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20058309599999999 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.649628+0000", 2024-09-16T04:42:14.040 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9387e-05 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.4201.0:209 1.0 e32/23 1:30306672:devicehealth::main.db.0000000000000000:head v 32'155, mlcod=30'152)", 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 33.244185663000003, 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20243324500000001, 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.041 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752715+0000", 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752717+0000", 2024-09-16T04:42:14.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.66e-06 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752778+0000", 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1816000000000006e-05 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.752805+0000", 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6335000000000001e-05 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.043 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.852941+0000", 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100135847 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.953240+0000", 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10029893300000001 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.953275+0000", 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5091e-05 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.955059+0000", 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001784683 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:40.955148+0000", 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8880000000000003e-05 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.045 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:106 3.es0 3:741a05b8:::smithi16530038-36:head [append 0~589824,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.361454+0000", 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.635446387000002, 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30540123299999999, 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 106 2024-09-16T04:42:14.046 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.361454+0000", 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.361454+0000", 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.361459+0000", 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.313e-06 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362586+0000", 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001127617 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362619+0000", 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2814999999999997e-05 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362635+0000", 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5534999999999999e-05 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362791+0000", 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000156391 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.363200+0000", 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040885900000000002 2024-09-16T04:42:14.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.365567+0000", 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0023674949999999998 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.462956+0000", 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097388308000000007 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.666740+0000", 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20378439700000001 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.666856+0000", 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000115503 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:107 3.es0 3:741a05b8:::smithi16530038-36:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:14.051 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362668+0000", 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.634232562000001, 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30452001299999998, 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 107 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362668+0000", 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362668+0000", 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362670+0000", 2024-09-16T04:42:14.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.643e-06 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362680+0000", 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8460000000000003e-06 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362699+0000", 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9568999999999998e-05 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.054 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362709+0000", 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.367375+0000", 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0046667230000000002 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.367636+0000", 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000261155 2024-09-16T04:42:14.055 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.368933+0000", 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012968910000000001 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.462959+0000", 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094025238999999997 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.056 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.667053+0000", 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20409445500000001 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.667188+0000", 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013508300000000001 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:108 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:14.057 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.634132957999999, 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60644887800000002, 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 108 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.059 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362769+0000", 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.364e-06 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362776+0000", 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9789999999999996e-06 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362795+0000", 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8569000000000001e-05 2024-09-16T04:42:14.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.061 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.061 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362811+0000", 2024-09-16T04:42:14.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6084999999999999e-05 2024-09-16T04:42:14.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369499+0000", 2024-09-16T04:42:14.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066885180000000001 2024-09-16T04:42:14.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:14.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369745+0000", 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024616900000000002 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669288+0000", 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29954281100000002 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669425+0000", 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013708200000000001 2024-09-16T04:42:14.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.064 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.064 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.969217+0000", 2024-09-16T04:42:14.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29979130100000001 2024-09-16T04:42:14.064 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.064 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 36/32 ECSubWriteReply(tid=13, last_complete=36'8, committed=1, applied=1))", 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.668440+0000", 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.328460873000001, 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.299416507, 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.065 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668440+0000", 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668440+0000", 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668441+0000", 2024-09-16T04:42:14.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5999999999999999e-06 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668450+0000", 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2090000000000005e-06 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.668479+0000", 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9354999999999999e-05 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.768613+0000", 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100134073 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967795+0000", 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19918244600000001 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967856+0000", 2024-09-16T04:42:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0823999999999999e-05 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpReadReply(3.es0 36/32 ECSubReadReply(tid=14, attrs_read=0))", 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.670590+0000", 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.326310725999999, 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29786822099999999, 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670590+0000", 2024-09-16T04:42:14.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670590+0000", 2024-09-16T04:42:14.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670592+0000", 2024-09-16T04:42:14.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9530000000000002e-06 2024-09-16T04:42:14.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670618+0000", 2024-09-16T04:42:14.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6375000000000001e-05 2024-09-16T04:42:14.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670647+0000", 2024-09-16T04:42:14.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8296000000000001e-05 2024-09-16T04:42:14.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.074 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.670665+0000", 2024-09-16T04:42:14.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8308000000000002e-05 2024-09-16T04:42:14.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.968355+0000", 2024-09-16T04:42:14.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29769044500000003 2024-09-16T04:42:14.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.968458+0000", 2024-09-16T04:42:14.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010284400000000001 2024-09-16T04:42:14.076 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.076 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.076 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.es0 36/32 ECSubWriteReply(tid=13, last_complete=36'8, committed=1, applied=1))", 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.755072+0000", 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.241828989999998, 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21260926999999999, 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755072+0000", 2024-09-16T04:42:14.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755072+0000", 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755073+0000", 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5e-06 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755088+0000", 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5291999999999999e-05 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755111+0000", 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2582000000000002e-05 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.755130+0000", 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9165e-05 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967621+0000", 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21249069800000001 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967681+0000", 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0032999999999999e-05 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(osd.3.22:2 3.es0 3:7ee6b8f9:::smithi16530038-8:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.182014+0000", 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 25.814886529999999, 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.302278981, 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182014+0000", 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182014+0000", 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182017+0000", 2024-09-16T04:42:14.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2430000000000001e-06 2024-09-16T04:42:14.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.083 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.083 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.083 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182028+0000", 2024-09-16T04:42:14.083 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0322999999999999e-05 2024-09-16T04:42:14.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.084 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.084 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.084 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182047+0000", 2024-09-16T04:42:14.084 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9779000000000002e-05 2024-09-16T04:42:14.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182062+0000", 2024-09-16T04:42:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4965e-05 2024-09-16T04:42:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182177+0000", 2024-09-16T04:42:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000114394 2024-09-16T04:42:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.182346+0000", 2024-09-16T04:42:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000169085 2024-09-16T04:42:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.484293+0000", 2024-09-16T04:42:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30194719199999998 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpReadReply(3.es0 38/32 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 25.812536496, 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29996245500000002, 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184364+0000", 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184400+0000", 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5800999999999998e-05 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.184413+0000", 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3169e-05 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.483752+0000", 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29933919199999998 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.484327+0000", 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00057429300000000003 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:281 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:53.099542+0000", 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.897359071, 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30637049, 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 281 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099542+0000", 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099542+0000", 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099543+0000", 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.569999999999999e-07 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099550+0000", 2024-09-16T04:42:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4189999999999996e-06 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099565+0000", 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4683e-05 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.099577+0000", 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2113000000000001e-05 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.203298+0000", 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103721174 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.203433+0000", 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000134764 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.405912+0000", 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20247937999999999 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs1 48/32 ECSubWrite(tid=75, reqid=client.4728.0:356, at_version=48'21, trim_to=0'0, roll_forward_to=48'19))", 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.460434+0000", 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.53646681, 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20368135400000001, 2024-09-16T04:42:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.460434+0000", 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.460434+0000", 2024-09-16T04:42:14.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.460435+0000", 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4839999999999999e-06 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.460471+0000", 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5932000000000001e-05 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.460498+0000", 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6712000000000001e-05 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.560589+0000", 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100091021 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.660766+0000", 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10017717900000001 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.660783+0000", 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6327000000000001e-05 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.664074+0000", 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0032910019999999999 2024-09-16T04:42:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.664115+0000", 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1696999999999997e-05 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:365 3.es0 3:741a05b8:::smithi16530038-36:head [create,setxattr YsO`TUrrXaTptX`eXscONOt_]dcP^Xa]S^TWWiqm (11788) in=40b,setxattr oi^cYciifcdaOrObmV`rpY]_[\\_XRNasd^fYcUiQ (1017) in=40b,setxattr NnlWipS`S[obrmTjfOrXNR_U^RaXm^RTUZSjeTPn (1683) in=40b,setxattr `Rdq]pmXkmok]RYTVc\\kl_fsshipieap^dmiSalQ (9120) in=40b,setxattr knfoakkVZXZZ]kbe_aTg^SsRaeWmatrnY_XYmNTW (19016) in=40b,setxattr scTN] (490) in=5b,setxattr PO]ZkWnlNifWZXRn_chZPacZ^ecXSeS]XbpaUedS (1366) in=40b,setxattr iblrYV^QO]RQb (490) in=13b,setxattr PhO_UPjl^dXjhPPlrpqkqVeccWOOR_a_`_]jrTio (1507) in=40b,setxattr UZTrkNZltdZkQ]toZYlWWldlYaqUeeaUptZhU`jb (876) in=40b,setxattr \\gmX]bkoXbYR[m\\f]gtseQ]SjNsrN]bQcaYflTOm (526) in=40b,setxattr UP_asas]_aYSd^sUrb[^eUm]neUY\\gtgcqWqipQ` (1226) in=40b,setxattr RbnfoadlmU (490) in=10b,setxattr jXdNonknr[NtSaQ\\pNdNYmVQcc]SoYrrom\\WnZPa (1122) in=40b,setxattr ]oZ_lqdXiNU`nUS\\^_nlTisrNtadtmge[^VY[rce (1086) in=40b,setxattr RcmOsXpXhRW\\o_cOt\\W_b[OZQahkUpfe]i\\k]rYs (1894) in=40b,setxattr YmQiO^S[Ok\\XN (490) in=13b,setxattr jeqjjZWTOTSkPsbNo\\]tYekhQcfX`_T\\_jpUQsTg (1295) in=40b,setxattr pYXdVeXTn`gtW`QWoQac^OYrt]kicggO[\\katbaZ (1507) in=40b,setxattr UWQrTe]`QdQdW`Wc_^t`Zg^^hVc[VUa`liQo[NXO (1015) in=40b] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:59.166696+0000", 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.830204189, 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40577269700000002, 2024-09-16T04:42:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 365 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166696+0000", 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166696+0000", 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166699+0000", 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5330000000000001e-06 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166838+0000", 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013858800000000001 2024-09-16T04:42:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166899+0000", 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1329999999999997e-05 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.266993+0000", 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100094238 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.267155+0000", 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016149299999999999 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.267480+0000", 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032509299999999999 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.270800+0000", 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003320192 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.275086+0000", 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0042863980000000003 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.572282+0000", 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29719557400000002 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.572469+0000", 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018725799999999999 2024-09-16T04:42:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:377 3.es0 3:741a05b8:::smithi16530038-36:head [delete] snapc 6=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:59.815015+0000", 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.181886129, 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.25315490499999999, 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 377 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.815015+0000", 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.815015+0000", 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.815017+0000", 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6699999999999998e-06 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.815029+0000", 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.149e-05 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.815062+0000", 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3597999999999999e-05 2024-09-16T04:42:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.815078+0000", 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5650000000000001e-05 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.915358+0000", 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100280276 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.915650+0000", 2024-09-16T04:42:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00029149500000000001 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.917081+0000", 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001431465 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.918788+0000", 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001706391 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.067957+0000", 2024-09-16T04:42:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14916912500000001 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.068169+0000", 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000212745 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.116 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:14.118 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:14.111+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:14.119 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:42:14.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:42:14.121 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-16T04:42:13.258167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49365555,"num_objects":70,"num_object_clones":19,"num_object_copies":470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":1,"num_read":321,"num_read_kb":29803,"num_write":501,"num_write_kb":45334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":648,"ondisk_log_size":648,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":215,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":556380,"kb_used_data":94808,"kb_used_omap":24,"kb_used_meta":461223,"kb_avail":1509393060,"statfs":{"total":1546188226560,"available":1545618493440,"internally_reserved":0,"allocated":97083392,"data_stored":115313698,"data_compressed":1803914,"data_compressed_allocated":2502656,"data_compressed_original":5865472,"omap_allocated":25522,"internal_metadata":472292430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,12,3,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-131072,"num_objects":-21,"num_object_clones":-16,"num_object_copies":-147,"num_objects_missing_on_primary":-17,"num_objects_missing":-17,"num_objects_degraded":-86,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":-2,"num_read":10,"num_read_kb":9507,"num_write":29,"num_write_kb":4544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":8323072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.023725"},"pg_stats":[{"pgid":"3.19","version":"50'30","reported_seq":189,"reported_epoch":59,"state":"activating+remapped","last_fresh":"2024-09-16T04:42:09.554546+0000","last_change":"2024-09-16T04:42:09.342748+0000","last_active":"2024-09-16T04:42:08.231676+0000","last_peered":"2024-09-16T04:42:06.828625+0000","last_clean":"2024-09-16T04:42:06.828625+0000","last_became_active":"2024-09-16T04:42:06.726227+0000","last_became_peered":"2024-09-16T04:42:06.726227+0000","last_unstale":"2024-09-16T04:42:09.554546+0000","last_undegraded":"2024-09-16T04:42:09.554546+0000","last_fullsized":"2024-09-16T04:42:09.554546+0000","mapping_epoch":58,"log_start":"44'22","ondisk_log_start":"44'22","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.098972921000000005,"stat_sum":{"num_bytes":1949696,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[14,1,0,3,15,8,10],"avail_no_missing":["14(0)","0(2)","1(1)","3(3)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"0(2),1(1),3(3),8(5),10(6),14(0),15(4)","objects":4}],"blocked_by":[],"up_primary":8,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"1"}]},{"pgid":"3.18","version":"44'8","reported_seq":169,"reported_epoch":59,"state":"activating+remapped","last_fresh":"2024-09-16T04:42:09.760253+0000","last_change":"2024-09-16T04:42:09.241696+0000","last_active":"2024-09-16T04:42:08.231689+0000","last_peered":"2024-09-16T04:42:07.041226+0000","last_clean":"2024-09-16T04:42:07.041226+0000","last_became_active":"2024-09-16T04:42:06.634893+0000","last_became_peered":"2024-09-16T04:42:06.634893+0000","last_unstale":"2024-09-16T04:42:09.760253+0000","last_undegraded":"2024-09-16T04:42:09.760253+0000","last_fullsized":"2024-09-16T04:42:09.760253+0000","mapping_epoch":58,"log_start":"44'8","ondisk_log_start":"44'8","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"38'4","last_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.115829+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0.203233094,"stat_sum":{"num_bytes":507904,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1129,"num_write":1,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,0,4,5,7,12],"acting":[5,0,2,1,14,15,6],"avail_no_missing":["5(0)","0(1)","1(3)","2(2)","6(6)","14(4)","15(5)"],"object_location_counts":[{"shards":"0(1),1(3),2(2),5(0),6(6),14(4),15(5)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"1"}]},{"pgid":"3.17","version":"44'6","reported_seq":62,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.240273+0000","last_change":"2024-09-16T04:42:09.241765+0000","last_active":"2024-09-16T04:42:09.240273+0000","last_peered":"2024-09-16T04:42:09.240273+0000","last_clean":"2024-09-16T04:42:09.240273+0000","last_became_active":"2024-09-16T04:42:04.604172+0000","last_became_peered":"2024-09-16T04:42:04.604172+0000","last_unstale":"2024-09-16T04:42:09.240273+0000","last_undegraded":"2024-09-16T04:42:09.240273+0000","last_fullsized":"2024-09-16T04:42:09.240273+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'6","last_scrub_stamp":"2024-09-16T04:42:00.511037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.511037+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021274789999999998,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1277952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.16","version":"35'2","reported_seq":66,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:42:12.384960+0000","last_change":"2024-09-16T04:42:10.385796+0000","last_active":"2024-09-16T04:42:12.384960+0000","last_peered":"2024-09-16T04:42:12.384960+0000","last_clean":"2024-09-16T04:42:12.384960+0000","last_became_active":"2024-09-16T04:42:04.405792+0000","last_became_peered":"2024-09-16T04:42:04.405792+0000","last_unstale":"2024-09-16T04:42:12.384960+0000","last_undegraded":"2024-09-16T04:42:12.384960+0000","last_fullsized":"2024-09-16T04:42:12.384960+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:57.977558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:57.977558+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:12.221273+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.0018703459999999999,"stat_sum":{"num_bytes":81920,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"3.15","version":"36'6","reported_seq":77,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:10.569758+0000","last_change":"2024-09-16T04:42:10.362585+0000","last_active":"2024-09-16T04:42:10.569758+0000","last_peered":"2024-09-16T04:42:10.569758+0000","last_clean":"2024-09-16T04:42:10.569758+0000","last_became_active":"2024-09-16T04:42:05.419352+0000","last_became_peered":"2024-09-16T04:42:05.419352+0000","last_unstale":"2024-09-16T04:42:10.569758+0000","last_undegraded":"2024-09-16T04:42:10.569758+0000","last_fullsized":"2024-09-16T04:42:10.569758+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020668259999999999,"stat_sum":{"num_bytes":761856,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.14","version":"59'36","reported_seq":93,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.452637+0000","last_change":"2024-09-16T04:42:09.455000+0000","last_active":"2024-09-16T04:42:09.452637+0000","last_peered":"2024-09-16T04:42:09.452637+0000","last_clean":"2024-09-16T04:42:09.452637+0000","last_became_active":"2024-09-16T04:42:04.626975+0000","last_became_peered":"2024-09-16T04:42:04.626975+0000","last_unstale":"2024-09-16T04:42:09.452637+0000","last_undegraded":"2024-09-16T04:42:09.452637+0000","last_fullsized":"2024-09-16T04:42:09.452637+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:05.952195+0000","scrub_duration":899,"objects_trimmed":0,"snaptrim_duration":0.11006980099999999,"stat_sum":{"num_bytes":3555328,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4423680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":81,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.247314+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:42:09.247314+0000","last_peered":"2024-09-16T04:42:09.247314+0000","last_clean":"2024-09-16T04:42:09.247314+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:42:09.247314+0000","last_undegraded":"2024-09-16T04:42:09.247314+0000","last_fullsized":"2024-09-16T04:42:09.247314+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:42:12.525664+0000","last_change":"2024-09-16T04:42:10.355085+0000","last_active":"2024-09-16T04:42:12.525664+0000","last_peered":"2024-09-16T04:42:12.525664+0000","last_clean":"2024-09-16T04:42:12.525664+0000","last_became_active":"2024-09-16T04:41:48.483004+0000","last_became_peered":"2024-09-16T04:41:48.483004+0000","last_unstale":"2024-09-16T04:42:12.525664+0000","last_undegraded":"2024-09-16T04:42:12.525664+0000","last_fullsized":"2024-09-16T04:42:12.525664+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:05.153670+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.00069638499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.339813+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:42:09.339813+0000","last_peered":"2024-09-16T04:42:09.339813+0000","last_clean":"2024-09-16T04:42:09.339813+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:42:09.339813+0000","last_undegraded":"2024-09-16T04:42:09.339813+0000","last_fullsized":"2024-09-16T04:42:09.339813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":139,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.453902+0000","last_change":"2024-09-16T04:42:09.454502+0000","last_active":"2024-09-16T04:42:09.453902+0000","last_peered":"2024-09-16T04:42:09.453902+0000","last_clean":"2024-09-16T04:42:09.453902+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:42:09.453902+0000","last_undegraded":"2024-09-16T04:42:09.453902+0000","last_fullsized":"2024-09-16T04:42:09.453902+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:42:00.916431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.916431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:13.788012+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.10661314,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.240113+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:42:09.240113+0000","last_peered":"2024-09-16T04:42:09.240113+0000","last_clean":"2024-09-16T04:42:09.240113+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:42:09.240113+0000","last_undegraded":"2024-09-16T04:42:09.240113+0000","last_fullsized":"2024-09-16T04:42:09.240113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"36'10","reported_seq":158,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.240349+0000","last_change":"2024-09-16T04:42:09.241366+0000","last_active":"2024-09-16T04:42:09.240349+0000","last_peered":"2024-09-16T04:42:09.240349+0000","last_clean":"2024-09-16T04:42:09.240349+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:42:09.240349+0000","last_undegraded":"2024-09-16T04:42:09.240349+0000","last_fullsized":"2024-09-16T04:42:09.240349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:59.997918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.997918+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:05.394987+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.0013049940000000001,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":80,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.247668+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:42:09.247668+0000","last_peered":"2024-09-16T04:42:09.247668+0000","last_clean":"2024-09-16T04:42:09.247668+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:42:09.247668+0000","last_undegraded":"2024-09-16T04:42:09.247668+0000","last_fullsized":"2024-09-16T04:42:09.247668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:26.420828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"57'21","reported_seq":168,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.247919+0000","last_change":"2024-09-16T04:42:09.248764+0000","last_active":"2024-09-16T04:42:09.247919+0000","last_peered":"2024-09-16T04:42:09.247919+0000","last_clean":"2024-09-16T04:42:09.247919+0000","last_became_active":"2024-09-16T04:41:48.475648+0000","last_became_peered":"2024-09-16T04:41:48.475648+0000","last_unstale":"2024-09-16T04:42:09.247919+0000","last_undegraded":"2024-09-16T04:42:09.247919+0000","last_fullsized":"2024-09-16T04:42:09.247919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:22.419260+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.00130339,"stat_sum":{"num_bytes":3817472,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":674,"num_write":1,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.3","version":"60'17","reported_seq":162,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:42:10.670834+0000","last_change":"2024-09-16T04:42:10.672971+0000","last_active":"2024-09-16T04:42:10.670834+0000","last_peered":"2024-09-16T04:42:10.670834+0000","last_clean":"2024-09-16T04:42:10.670834+0000","last_became_active":"2024-09-16T04:41:48.281236+0000","last_became_peered":"2024-09-16T04:41:48.281236+0000","last_unstale":"2024-09-16T04:42:10.670834+0000","last_undegraded":"2024-09-16T04:42:10.670834+0000","last_fullsized":"2024-09-16T04:42:10.670834+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:41:49.293927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.293927+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:06.960029+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.309867214,"stat_sum":{"num_bytes":2539520,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.346108+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:42:09.346108+0000","last_peered":"2024-09-16T04:42:09.346108+0000","last_clean":"2024-09-16T04:42:09.346108+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:42:09.346108+0000","last_undegraded":"2024-09-16T04:42:09.346108+0000","last_fullsized":"2024-09-16T04:42:09.346108+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":141,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.347046+0000","last_change":"2024-09-16T04:42:09.447382+0000","last_active":"2024-09-16T04:42:09.347046+0000","last_peered":"2024-09-16T04:42:09.347046+0000","last_clean":"2024-09-16T04:42:09.347046+0000","last_became_active":"2024-09-16T04:41:48.383143+0000","last_became_peered":"2024-09-16T04:41:48.383143+0000","last_unstale":"2024-09-16T04:42:09.347046+0000","last_undegraded":"2024-09-16T04:42:09.347046+0000","last_fullsized":"2024-09-16T04:42:09.347046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:49.599894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:49.599894+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:01.482265+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0.100838002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:42:10.362462+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:42:10.362462+0000","last_peered":"2024-09-16T04:42:10.362462+0000","last_clean":"2024-09-16T04:42:10.362462+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:42:10.362462+0000","last_undegraded":"2024-09-16T04:42:10.362462+0000","last_fullsized":"2024-09-16T04:42:10.362462+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"59'29","reported_seq":177,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.452794+0000","last_change":"2024-09-16T04:42:09.453762+0000","last_active":"2024-09-16T04:42:09.452794+0000","last_peered":"2024-09-16T04:42:09.452794+0000","last_clean":"2024-09-16T04:42:09.452794+0000","last_became_active":"2024-09-16T04:41:48.404175+0000","last_became_peered":"2024-09-16T04:41:48.404175+0000","last_unstale":"2024-09-16T04:42:09.452794+0000","last_undegraded":"2024-09-16T04:42:09.452794+0000","last_fullsized":"2024-09-16T04:42:09.452794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:50.183380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.183380+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:50.245980+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0.206389041,"stat_sum":{"num_bytes":2867200,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":1875,"num_write":10,"num_write_kb":2800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":81,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.344167+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:42:09.344167+0000","last_peered":"2024-09-16T04:42:09.344167+0000","last_clean":"2024-09-16T04:42:09.344167+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:42:09.344167+0000","last_undegraded":"2024-09-16T04:42:09.344167+0000","last_fullsized":"2024-09-16T04:42:09.344167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"59'15","reported_seq":160,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:42:10.363557+0000","last_change":"2024-09-16T04:42:10.365012+0000","last_active":"2024-09-16T04:42:10.363557+0000","last_peered":"2024-09-16T04:42:10.363557+0000","last_clean":"2024-09-16T04:42:10.363557+0000","last_became_active":"2024-09-16T04:41:48.477867+0000","last_became_peered":"2024-09-16T04:41:48.477867+0000","last_unstale":"2024-09-16T04:42:10.363557+0000","last_undegraded":"2024-09-16T04:42:10.363557+0000","last_fullsized":"2024-09-16T04:42:10.363557+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.564481+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:14.470803+0000","scrub_duration":1083,"objects_trimmed":0,"snaptrim_duration":0.0020386699999999998,"stat_sum":{"num_bytes":2048000,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":82,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.240546+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:42:09.240546+0000","last_peered":"2024-09-16T04:42:09.240546+0000","last_clean":"2024-09-16T04:42:09.240546+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:42:09.240546+0000","last_undegraded":"2024-09-16T04:42:09.240546+0000","last_fullsized":"2024-09-16T04:42:09.240546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":277,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:42:10.361838+0000","last_change":"2024-09-16T04:41:26.345501+0000","last_active":"2024-09-16T04:42:10.361838+0000","last_peered":"2024-09-16T04:42:10.361838+0000","last_clean":"2024-09-16T04:42:10.361838+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:42:10.361838+0000","last_undegraded":"2024-09-16T04:42:10.361838+0000","last_fullsized":"2024-09-16T04:42:10.361838+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:25.800091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"57'10","reported_seq":174,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.240072+0000","last_change":"2024-09-16T04:42:09.248731+0000","last_active":"2024-09-16T04:42:09.240072+0000","last_peered":"2024-09-16T04:42:09.240072+0000","last_clean":"2024-09-16T04:42:09.240072+0000","last_became_active":"2024-09-16T04:42:06.535331+0000","last_became_peered":"2024-09-16T04:42:06.535331+0000","last_unstale":"2024-09-16T04:42:09.240072+0000","last_undegraded":"2024-09-16T04:42:09.240072+0000","last_fullsized":"2024-09-16T04:42:09.240072+0000","mapping_epoch":32,"log_start":"44'8","ondisk_log_start":"44'8","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'4","last_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.115829+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":12,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0.0091626309999999992,"stat_sum":{"num_bytes":262144,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1128,"num_write":0,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.9","version":"50'30","reported_seq":190,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.445945+0000","last_change":"2024-09-16T04:42:09.556218+0000","last_active":"2024-09-16T04:42:09.445945+0000","last_peered":"2024-09-16T04:42:09.445945+0000","last_clean":"2024-09-16T04:42:09.445945+0000","last_became_active":"2024-09-16T04:42:06.419535+0000","last_became_peered":"2024-09-16T04:42:06.419535+0000","last_unstale":"2024-09-16T04:42:09.445945+0000","last_undegraded":"2024-09-16T04:42:09.445945+0000","last_fullsized":"2024-09-16T04:42:09.445945+0000","mapping_epoch":32,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.110618728,"stat_sum":{"num_bytes":1949696,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.a","version":"57'23","reported_seq":161,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.346166+0000","last_change":"2024-09-16T04:42:09.347108+0000","last_active":"2024-09-16T04:42:09.346166+0000","last_peered":"2024-09-16T04:42:09.346166+0000","last_clean":"2024-09-16T04:42:09.346166+0000","last_became_active":"2024-09-16T04:41:42.059056+0000","last_became_peered":"2024-09-16T04:41:42.059056+0000","last_unstale":"2024-09-16T04:42:09.346166+0000","last_undegraded":"2024-09-16T04:42:09.346166+0000","last_fullsized":"2024-09-16T04:42:09.346166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.868636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00141366,"stat_sum":{"num_bytes":507904,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":3273,"num_write":33,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.b","version":"59'33","reported_seq":160,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.655085+0000","last_change":"2024-09-16T04:42:09.656869+0000","last_active":"2024-09-16T04:42:09.655085+0000","last_peered":"2024-09-16T04:42:09.655085+0000","last_clean":"2024-09-16T04:42:09.655085+0000","last_became_active":"2024-09-16T04:41:42.059548+0000","last_became_peered":"2024-09-16T04:41:42.059548+0000","last_unstale":"2024-09-16T04:42:09.655085+0000","last_undegraded":"2024-09-16T04:42:09.655085+0000","last_fullsized":"2024-09-16T04:42:09.655085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:09.558082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30907998199999998,"stat_sum":{"num_bytes":4505600,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":1929,"num_write":36,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.c","version":"60'52","reported_seq":183,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:42:12.384922+0000","last_change":"2024-09-16T04:42:10.994980+0000","last_active":"2024-09-16T04:42:12.384922+0000","last_peered":"2024-09-16T04:42:12.384922+0000","last_clean":"2024-09-16T04:42:12.384922+0000","last_became_active":"2024-09-16T04:41:42.057674+0000","last_became_peered":"2024-09-16T04:41:42.057674+0000","last_unstale":"2024-09-16T04:42:12.384922+0000","last_undegraded":"2024-09-16T04:42:12.384922+0000","last_fullsized":"2024-09-16T04:42:12.384922+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:50.936582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51109875500000002,"stat_sum":{"num_bytes":3375104,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":1544,"num_write":80,"num_write_kb":4256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"3.d","version":"57'89","reported_seq":222,"reported_epoch":59,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:09.238164+0000","last_change":"2024-09-16T04:42:09.238210+0000","last_active":"2024-09-16T04:42:09.238164+0000","last_peered":"2024-09-16T04:42:09.238164+0000","last_clean":"2024-09-16T04:42:09.238164+0000","last_became_active":"2024-09-16T04:41:42.053696+0000","last_became_peered":"2024-09-16T04:41:42.053696+0000","last_unstale":"2024-09-16T04:42:09.238164+0000","last_undegraded":"2024-09-16T04:42:09.238164+0000","last_fullsized":"2024-09-16T04:42:09.238164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:47.022603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35223036400000002,"stat_sum":{"num_bytes":9191424,"num_objects":9,"num_object_clones":3,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":39,"num_read_kb":12303,"num_write":45,"num_write_kb":11216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"1"}]},{"pgid":"3.e","version":"57'27","reported_seq":157,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T04:42:09.250863+0000","last_change":"2024-09-16T04:42:09.251843+0000","last_active":"2024-09-16T04:42:09.250863+0000","last_peered":"2024-09-16T04:42:09.250863+0000","last_clean":"2024-09-16T04:42:09.250863+0000","last_became_active":"2024-09-16T04:41:42.158174+0000","last_became_peered":"2024-09-16T04:41:42.158174+0000","last_unstale":"2024-09-16T04:42:09.250863+0000","last_undegraded":"2024-09-16T04:42:09.250863+0000","last_fullsized":"2024-09-16T04:42:09.250863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:53.687598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014823429999999999,"stat_sum":{"num_bytes":2473984,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":4088,"num_write":36,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.f","version":"59'43","reported_seq":181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:42:12.385169+0000","last_change":"2024-09-16T04:42:10.384667+0000","last_active":"2024-09-16T04:42:12.385169+0000","last_peered":"2024-09-16T04:42:12.385169+0000","last_clean":"2024-09-16T04:42:12.385169+0000","last_became_active":"2024-09-16T04:41:41.950892+0000","last_became_peered":"2024-09-16T04:41:41.950892+0000","last_unstale":"2024-09-16T04:42:12.385169+0000","last_undegraded":"2024-09-16T04:42:12.385169+0000","last_fullsized":"2024-09-16T04:42:12.385169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:09.480165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080018699999999995,"stat_sum":{"num_bytes":1785856,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":1735,"num_write":18,"num_write_kb":2896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"3.10","version":"44'10","reported_seq":77,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:10.369475+0000","last_change":"2024-09-16T04:42:10.404555+0000","last_active":"2024-09-16T04:42:10.369475+0000","last_peered":"2024-09-16T04:42:10.369475+0000","last_clean":"2024-09-16T04:42:10.369475+0000","last_became_active":"2024-09-16T04:42:04.808967+0000","last_became_peered":"2024-09-16T04:42:04.808967+0000","last_unstale":"2024-09-16T04:42:10.369475+0000","last_undegraded":"2024-09-16T04:42:10.369475+0000","last_fullsized":"2024-09-16T04:42:10.369475+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:41:59.373567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.373567+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075603650000000003,"stat_sum":{"num_bytes":655360,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,0,11],"acting":[13,6,7,4,8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"3.11","version":"57'28","reported_seq":93,"reported_epoch":59,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:09.653787+0000","last_change":"2024-09-16T04:42:09.302374+0000","last_active":"2024-09-16T04:42:09.653787+0000","last_peered":"2024-09-16T04:42:09.653787+0000","last_clean":"2024-09-16T04:42:09.653787+0000","last_became_active":"2024-09-16T04:42:04.623974+0000","last_became_peered":"2024-09-16T04:42:04.623974+0000","last_unstale":"2024-09-16T04:42:09.653787+0000","last_undegraded":"2024-09-16T04:42:09.653787+0000","last_fullsized":"2024-09-16T04:42:09.653787+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'24","last_scrub_stamp":"2024-09-16T04:41:58.297098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.297098+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046599299999999996,"stat_sum":{"num_bytes":3465216,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":13,"num_write_kb":3384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4784128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"}]},{"pgid":"3.12","version":"57'12","reported_seq":76,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T04:42:10.566378+0000","last_change":"2024-09-16T04:42:10.568007+0000","last_active":"2024-09-16T04:42:10.566378+0000","last_peered":"2024-09-16T04:42:10.566378+0000","last_clean":"2024-09-16T04:42:10.566378+0000","last_became_active":"2024-09-16T04:42:04.406048+0000","last_became_peered":"2024-09-16T04:42:04.406048+0000","last_unstale":"2024-09-16T04:42:10.566378+0000","last_undegraded":"2024-09-16T04:42:10.566378+0000","last_fullsized":"2024-09-16T04:42:10.566378+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'12","last_scrub_stamp":"2024-09-16T04:42:10.115913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.115913+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:22.683479+0000","scrub_duration":611,"objects_trimmed":0,"snaptrim_duration":0.0021406680000000001,"stat_sum":{"num_bytes":1277952,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]},{"pgid":"3.13","version":"36'3","reported_seq":63,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T04:42:12.515004+0000","last_change":"2024-09-16T04:42:10.693013+0000","last_active":"2024-09-16T04:42:12.515004+0000","last_peered":"2024-09-16T04:42:12.515004+0000","last_clean":"2024-09-16T04:42:12.515004+0000","last_became_active":"2024-09-16T04:42:04.415210+0000","last_became_peered":"2024-09-16T04:42:04.415210+0000","last_unstale":"2024-09-16T04:42:12.515004+0000","last_undegraded":"2024-09-16T04:42:12.515004+0000","last_fullsized":"2024-09-16T04:42:12.515004+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:42:10.690368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.690368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:14.617473+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.30916923299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,0,4,10],"acting":[11,7,9,13,0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"5"},{"start":"8","length":"2"},{"start":"c","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":48775168,"num_objects":66,"num_object_clones":19,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":1,"num_read":185,"num_read_kb":29685,"num_write":296,"num_write_kb":41872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":11878400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":85897216,"data_stored":105971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":488,"ondisk_log_size":488,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247822,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36580,"kb_used_data":5384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335260,"statfs":{"total":96636764160,"available":96599306240,"internally_reserved":0,"allocated":5513216,"data_stored":6248690,"data_compressed":143945,"data_compressed_allocated":196608,"data_compressed_original":823296,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247822,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36612,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335228,"statfs":{"total":96636764160,"available":96599273472,"internally_reserved":0,"allocated":5550080,"data_stored":6424274,"data_compressed":112000,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247822,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37372,"kb_used_data":6176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334468,"statfs":{"total":96636764160,"available":96598495232,"internally_reserved":0,"allocated":6324224,"data_stored":7915218,"data_compressed":112000,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313229,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39188,"kb_used_data":7992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332652,"statfs":{"total":96636764160,"available":96596635648,"internally_reserved":0,"allocated":8183808,"data_stored":10835867,"data_compressed":118050,"data_compressed_allocated":163840,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313229,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":5084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":5206016,"data_stored":5877569,"data_compressed":115027,"data_compressed_allocated":159744,"data_compressed_original":319488,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280525,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30908,"kb_used_data":3884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340932,"statfs":{"total":96636764160,"available":96605114368,"internally_reserved":0,"allocated":3977216,"data_stored":5199740,"data_compressed":137840,"data_compressed_allocated":188416,"data_compressed_original":806912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32952,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":6066176,"data_stored":6872599,"data_compressed":108946,"data_compressed_allocated":151552,"data_compressed_original":303104,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280525,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33424,"kb_used_data":6400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338416,"statfs":{"total":96636764160,"available":96602537984,"internally_reserved":0,"allocated":6553600,"data_stored":7460207,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280525,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39948,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331892,"statfs":{"total":96636764160,"available":96595857408,"internally_reserved":0,"allocated":8970240,"data_stored":10650972,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,8,3,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280525,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31044,"kb_used_data":4020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340796,"statfs":{"total":96636764160,"available":96604975104,"internally_reserved":0,"allocated":4116480,"data_stored":4613468,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280524,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":6424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":6578176,"data_stored":7734620,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37616,"kb_used_data":6428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334224,"statfs":{"total":96636764160,"available":96598245376,"internally_reserved":0,"allocated":6582272,"data_stored":7738716,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35256,"kb_used_data":8232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336584,"statfs":{"total":96636764160,"available":96600662016,"internally_reserved":0,"allocated":8429568,"data_stored":9565532,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280525,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33884,"kb_used_data":6856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337956,"statfs":{"total":96636764160,"available":96602066944,"internally_reserved":0,"allocated":7020544,"data_stored":8461847,"data_compressed":108946,"data_compressed_allocated":151552,"data_compressed_original":303104,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30128,"kb_used_data":3104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341712,"statfs":{"total":96636764160,"available":96605913088,"internally_reserved":0,"allocated":3178496,"data_stored":4150639,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280525,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35908,"kb_used_data":4720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335932,"statfs":{"total":96636764160,"available":96599994368,"internally_reserved":0,"allocated":4833280,"data_stored":5563740,"data_compressed":105895,"data_compressed_allocated":147456,"data_compressed_original":294912,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7684096,"data_stored":10559488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5656576,"data_stored":7467008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3272704,"data_stored":3760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5906432,"data_stored":7032832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7438336,"data_stored":8634368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3473408,"data_stored":4186112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":7307264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5939200,"data_stored":7147520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":7843840,"data_stored":9515008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4882432,"data_stored":5976064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6365184,"data_stored":8024064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4526080,"data_stored":5419008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2588672,"data_stored":3780608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4784128,"data_stored":5271552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4190208,"data_stored":5455872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":6434816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:14.123 INFO:tasks.thrashosds.thrasher:Setting 3.10 to [2, 7, 1, 14, 5, 3, 15] 2024-09-16T04:42:14.123 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.10', '2', '7', '1', '14', '5', '3', '15'] 2024-09-16T04:42:14.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.10 2 7 1 14 5 3 15 2024-09-16T04:42:14.218 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:14.216+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:14.250 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:14.250 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:14.250 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:14.251 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:14.263 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:42:14.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:42:14.271 INFO:tasks.rados.rados.0.smithi165.stdout:190: done (6 left) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:191: done (5 left) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:193: done (4 left) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:194: done (3 left) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:196: done (2 left) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:199: done (1 left) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:201: done (0 left) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:202: copy_from oid 17 from oid 8 current snap is 15 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:203: read oid 32 snap -1 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:203: expect (ObjNum 55 snap 1 seq_num 55) 2024-09-16T04:42:14.272 INFO:tasks.rados.rados.0.smithi165.stdout:204: snap_remove snap 14 2024-09-16T04:42:14.319 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:14.316+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:14.391 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:14.391 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:14.391 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:14.391 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:14.392 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:14.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:42:14.450 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:14.448+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:14.520 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:14.516+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 36/32 ECSubWrite(tid=6, reqid=client.4728.0:118, at_version=36'5, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.756468+0000", 2024-09-16T04:42:14.578 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.816912254000002, 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102545397, 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756468+0000", 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756468+0000", 2024-09-16T04:42:14.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756469+0000", 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6300000000000004e-07 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756729+0000", 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025997800000000001 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.756748+0000", 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9018999999999999e-05 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.856849+0000", 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010113399999999 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.857149+0000", 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00030028299999999998 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.857163+0000", 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3752e-05 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.858932+0000", 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017691759999999999 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.859013+0000", 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1291999999999993e-05 2024-09-16T04:42:14.582 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/32 ECSubWrite(tid=161, reqid=client.4728.0:127, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.862409+0000", 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.710971180000001, 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10371385299999999, 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.862409+0000", 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.862409+0000", 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.862411+0000", 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4600000000000002e-06 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.863037+0000", 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00062557300000000004 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.863071+0000", 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4214999999999997e-05 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.863088+0000", 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7299000000000002e-05 2024-09-16T04:42:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.963347+0000", 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100258786 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.963372+0000", 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5168999999999999e-05 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.966059+0000", 2024-09-16T04:42:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026868450000000002 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.966123+0000", 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3505999999999998e-05 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.as5 36/32 ECSubWrite(tid=16, reqid=client.4728.0:171, at_version=36'9, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.898522+0000", 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.674858291, 2024-09-16T04:42:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10165025799999999, 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898522+0000", 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898522+0000", 2024-09-16T04:42:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898524+0000", 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.57e-06 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898566+0000", 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1284e-05 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898600+0000", 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3940999999999999e-05 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.898619+0000", 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9289000000000001e-05 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.998828+0000", 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100208912 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.998851+0000", 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2838000000000001e-05 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.000097+0000", 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012462020000000001 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.000172+0000", 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5222000000000006e-05 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 38/32 ECSubWrite(tid=34, reqid=client.4728.0:182, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.276943+0000", 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.296436635999999, 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10333068199999999, 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.276943+0000", 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.276943+0000", 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.276945+0000", 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3799999999999999e-06 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.277066+0000", 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0001209 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.277104+0000", 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8708000000000003e-05 2024-09-16T04:42:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.277129+0000", 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4085999999999999e-05 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.377334+0000", 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100205586 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.377349+0000", 2024-09-16T04:42:14.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4666e-05 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.380172+0000", 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0028237119999999999 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.380274+0000", 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000101644 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 38/32 ECSubWrite(tid=35, reqid=client.4728.0:183, at_version=38'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.279231+0000", 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.294149252, 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102482004, 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279231+0000", 2024-09-16T04:42:14.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279231+0000", 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279232+0000", 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.593e-06 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.598 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279264+0000", 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1535000000000002e-05 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279302+0000", 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7914999999999999e-05 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.279326+0000", 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4215000000000001e-05 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.379685+0000", 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100358959 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.379710+0000", 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4708000000000001e-05 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.381607+0000", 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001896752 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.381713+0000", 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000106327 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.163525734, 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20191694600000001, 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.601 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409856+0000", 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409917+0000", 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1023999999999997e-05 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409958+0000", 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0945000000000002e-05 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510088+0000", 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100130017 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510247+0000", 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015866599999999999 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510287+0000", 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0330999999999997e-05 2024-09-16T04:42:14.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611689+0000", 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101401328 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611771+0000", 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2582000000000003e-05 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPush(3.17s1 42/41 [PushOp(3:ed82f61a:::smithi16530038-1:head, version: 40'4, data_included: [0~151552], data_size: 151552, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed82f61a:::smithi16530038-1:head@40'4, size: 606208, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 606208, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.831178+0000", 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.742201751, 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102367006, 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831178+0000", 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831178+0000", 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831180+0000", 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.513e-06 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831696+0000", 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00051605900000000003 2024-09-16T04:42:14.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.831742+0000", 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5951000000000001e-05 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.931903+0000", 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.1001611 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.932153+0000", 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025006099999999999 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.933545+0000", 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013923220000000001 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=39, reqid=client.4728.0:187, at_version=0'0, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.923239+0000", 2024-09-16T04:42:14.609 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.650141262999998, 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101838922, 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923239+0000", 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923239+0000", 2024-09-16T04:42:14.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923241+0000", 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8139999999999999e-06 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923264+0000", 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3142e-05 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923340+0000", 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6095999999999998e-05 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.923398+0000", 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7883999999999997e-05 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.023680+0000", 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100282702 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.023695+0000", 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4656e-05 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.024995+0000", 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001299929 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:53.025078+0000", 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2699000000000002e-05 2024-09-16T04:42:14.613 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s6 44/37 ECSubWrite(tid=11, reqid=osd.5.0:11, at_version=44'8, trim_to=0'0, roll_forward_to=42'6))", 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.455280978000001, 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10484916599999999, 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118100+0000", 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9599999999999998e-07 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118155+0000", 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4663999999999997e-05 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118199+0000", 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4861000000000003e-05 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118225+0000", 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5415999999999999e-05 2024-09-16T04:42:14.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118379+0000", 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015390299999999999 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118406+0000", 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7317999999999998e-05 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222865+0000", 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10445916099999999 2024-09-16T04:42:14.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222948+0000", 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3047000000000006e-05 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 44/41 ECSubWrite(tid=40, reqid=osd.14.0:40, at_version=44'10, trim_to=0'0, roll_forward_to=42'8))", 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.448779438999999, 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10233370999999999, 2024-09-16T04:42:14.618 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124601+0000", 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8999999999999996e-07 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124656+0000", 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4641000000000001e-05 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.124686+0000", 2024-09-16T04:42:14.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9872000000000001e-05 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224794+0000", 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100107847 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224954+0000", 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000160163 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224980+0000", 2024-09-16T04:42:14.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6162000000000001e-05 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.226840+0000", 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001859793 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.226934+0000", 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4542000000000004e-05 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.622 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.12s3 44/41 ECSubWrite(tid=179, reqid=osd.13.0:179, at_version=44'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.352166211, 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103206719, 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:42:14.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221214+0000", 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4000000000000002e-07 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221253+0000", 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9118000000000001e-05 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221277+0000", 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3828999999999998e-05 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.221293+0000", 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5502000000000002e-05 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.321532+0000", 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100239168 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.321562+0000", 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9884999999999999e-05 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.324333+0000", 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027712380000000001 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.324421+0000", 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7539000000000003e-05 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.2s3 53/37 ECSubWrite(tid=192, reqid=client.4728.0:420, at_version=53'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:03.323696+0000", 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.249683664999999, 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102789378, 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.627 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.323696+0000", 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.323696+0000", 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.323699+0000", 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2230000000000001e-06 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.323963+0000", 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026460399999999999 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.323993+0000", 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9465e-05 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424151+0000", 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100158366 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424426+0000", 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027480000000000001 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424444+0000", 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8071999999999999e-05 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.426404+0000", 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019596980000000002 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.426486+0000", 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2150000000000005e-05 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.631 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.2s3 53/37 ECSubWrite(tid=193, reqid=client.4728.0:421, at_version=53'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:03.424201+0000", 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.149179519, 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10190629399999999, 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424201+0000", 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424201+0000", 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424202+0000", 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7000000000000003e-07 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424288+0000", 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6096999999999999e-05 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424312+0000", 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4494999999999998e-05 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.424326+0000", 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3363000000000001e-05 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.524414+0000", 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10008887800000001 2024-09-16T04:42:14.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.524429+0000", 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4619e-05 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.526027+0000", 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015981719999999999 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:03.526107+0000", 2024-09-16T04:42:14.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9699999999999999e-05 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 58/32 ECSubWrite(tid=118, reqid=client.4728.0:458, at_version=58'27, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.405078+0000", 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.1683018030000003, 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103927176, 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405078+0000", 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405078+0000", 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405080+0000", 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9300000000000002e-06 2024-09-16T04:42:14.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405351+0000", 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027029099999999998 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405381+0000", 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0571999999999998e-05 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405398+0000", 2024-09-16T04:42:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6595999999999998e-05 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405533+0000", 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013564600000000001 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405555+0000", 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2079e-05 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.508926+0000", 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103370634 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.509005+0000", 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9427999999999999e-05 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 58/32 ECSubWrite(tid=119, reqid=client.4728.0:459, at_version=58'28, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:14.640 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.406245+0000", 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.1671349319999997, 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10302246299999999, 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.406245+0000", 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.406245+0000", 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.406247+0000", 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7770000000000001e-06 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.406366+0000", 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000118663 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.406399+0000", 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3838999999999999e-05 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.406421+0000", 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1931999999999999e-05 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.507388+0000", 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10096622 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.507403+0000", 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5478999999999999e-05 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.509198+0000", 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001795392 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.509268+0000", 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9160999999999997e-05 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.644 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 58/32 ECSubWrite(tid=120, reqid=client.4728.0:460, at_version=58'29, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.407164+0000", 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.1662159079999999, 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102876917, 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.407164+0000", 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.407164+0000", 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.407166+0000", 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7999999999999999e-06 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.407204+0000", 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8028000000000003e-05 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.407234+0000", 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0419000000000001e-05 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.407256+0000", 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1746000000000002e-05 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.508458+0000", 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101202139 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.508482+0000", 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4116000000000001e-05 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.509963+0000", 2024-09-16T04:42:14.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014808810000000001 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.510041+0000", 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7787999999999995e-05 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 58/32 ECSubWrite(tid=116, reqid=client.4728.0:462, at_version=58'44, trim_to=0'0, roll_forward_to=57'42))", 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.836550+0000", 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.7368298700000002, 2024-09-16T04:42:14.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103108159, 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836550+0000", 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836550+0000", 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836551+0000", 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8299999999999998e-07 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836854+0000", 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00030358700000000002 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836884+0000", 2024-09-16T04:42:14.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9564999999999999e-05 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836900+0000", 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6489000000000001e-05 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.937191+0000", 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100290958 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.937217+0000", 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6020000000000002e-05 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.939583+0000", 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002365556 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.939658+0000", 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5600999999999997e-05 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.653 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.2s3 59/37 ECSubWrite(tid=207, reqid=osd.13.0:207, at_version=59'15, trim_to=0'0, roll_forward_to=57'13))", 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:09.359380+0000", 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.2140002140000004, 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101755926, 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.359380+0000", 2024-09-16T04:42:14.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.359380+0000", 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.359381+0000", 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0699999999999996e-07 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.359446+0000", 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5448000000000001e-05 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.359478+0000", 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1392e-05 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.459565+0000", 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100087335 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.459706+0000", 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014155600000000001 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.459730+0000", 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3913e-05 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.461055+0000", 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013250079999999999 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.461136+0000", 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0467000000000001e-05 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 59/32 ECSubWrite(tid=122, reqid=osd.11.0:119, at_version=0'0, trim_to=0'0, roll_forward_to=59'48))", 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:09.541436+0000", 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.0319437750000002, 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10170979199999999, 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.658 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.541436+0000", 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.541436+0000", 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.541438+0000", 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.007e-06 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.541462+0000", 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3739000000000001e-05 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.541492+0000", 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9609000000000001e-05 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.641629+0000", 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100137822 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.641828+0000", 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019892299999999999 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.641857+0000", 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8822e-05 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.643078+0000", 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012209090000000001 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.643146+0000", 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7960999999999995e-05 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.1019759010000003, 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246694400000001, 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471405+0000", 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471443+0000", 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7605000000000003e-05 2024-09-16T04:42:14.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471463+0000", 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0452999999999999e-05 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.571565+0000", 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100102105 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671828+0000", 2024-09-16T04:42:14.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026227 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671853+0000", 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5826e-05 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673802+0000", 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019481730000000001 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:14.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673871+0000", 2024-09-16T04:42:14.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9624999999999998e-05 2024-09-16T04:42:14.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.667 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:14.667 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:14.667 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:14.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:42:14.669 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:14.664+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:14.769 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:14.767+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:14.837 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:14.837 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:14.837 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:14.837 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:14.851 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:42:14.871 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:14.867+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:14.969 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:14.969 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:14.969 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:14.969 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:14.969 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:14.972 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:14.967+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:14.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:42:15.073 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:15.068+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:15.103 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:15.103 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:15.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:15.103 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:15.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpRead(3.cs3 36/32 ECSubRead(tid=24, to_read={3:372a9655:::smithi16530038-32:head=0,4096,0}, subchunks={3:372a9655:::smithi16530038-32:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.967149+0000", 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.13319353, 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.200811252, 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967149+0000", 2024-09-16T04:42:15.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967149+0000", 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967152+0000", 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0940000000000002e-06 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967169+0000", 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7549000000000001e-05 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967212+0000", 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2701000000000003e-05 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.967243+0000", 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1015e-05 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.167603+0000", 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20036040099999999 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.167961+0000", 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000357492 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.107 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=17, reqid=client.4728.0:139, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.013492156000002, 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.203994169, 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086853+0000", 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6000000000000001e-06 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087506+0000", 2024-09-16T04:42:15.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00065279900000000002 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087552+0000", 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5840999999999997e-05 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087581+0000", 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9354999999999999e-05 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187852+0000", 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100270242 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187866+0000", 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4286e-05 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290791+0000", 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102925166 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290845+0000", 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3879999999999999e-05 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=18, reqid=client.4728.0:140, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.010806751000001, 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27450581499999999, 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089538+0000", 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0370000000000001e-06 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089605+0000", 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6268999999999999e-05 2024-09-16T04:42:15.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089663+0000", 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8257000000000001e-05 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.189764+0000", 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100101525 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290404+0000", 2024-09-16T04:42:15.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10063934500000001 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290428+0000", 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4782000000000001e-05 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363982+0000", 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073553303 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364042+0000", 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0297000000000002e-05 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 36/32 ECSubWrite(tid=27, reqid=client.4728.0:136, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.162782+0000", 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.937560521999998, 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20118729299999999, 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.117 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.162782+0000", 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.162782+0000", 2024-09-16T04:42:15.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.162785+0000", 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163121+0000", 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00033597100000000001 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163151+0000", 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0787999999999997e-05 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163167+0000", 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6008999999999999e-05 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.188993+0000", 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.025825142999999998 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.189015+0000", 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2498999999999999e-05 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363899+0000", 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.17488398499999999 2024-09-16T04:42:15.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363970+0000", 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0702000000000002e-05 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 36/32 ECSubWrite(tid=29, reqid=client.4728.0:137, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.163851+0000", 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.936492414, 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20078317000000001, 2024-09-16T04:42:15.122 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163851+0000", 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163851+0000", 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163852+0000", 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7430000000000001e-06 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163889+0000", 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6275000000000001e-05 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163924+0000", 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5131e-05 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.163946+0000", 2024-09-16T04:42:15.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1829000000000001e-05 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363543+0000", 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19959737499999999 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363557+0000", 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4348999999999999e-05 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364556+0000", 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099826600000000004 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364634+0000", 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8201999999999994e-05 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.127 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/32 ECSubWrite(tid=32, reqid=client.4728.0:209, at_version=38'17, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.625281928, 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202405108, 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475063+0000", 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.906e-06 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475102+0000", 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9195e-05 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.130 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475132+0000", 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9861000000000001e-05 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475148+0000", 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.611e-05 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.575413+0000", 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026479000000001 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.575431+0000", 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7912000000000001e-05 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677395+0000", 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101964155 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677466+0000", 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1179000000000004e-05 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/32 ECSubWrite(tid=33, reqid=client.4728.0:210, at_version=38'18, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.475188+0000", 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.625154675000001, 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202351791, 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475188+0000", 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475188+0000", 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475189+0000", 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4600000000000004e-07 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475217+0000", 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8388999999999999e-05 2024-09-16T04:42:15.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475231+0000", 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3336000000000001e-05 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475243+0000", 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2245999999999999e-05 2024-09-16T04:42:15.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.576236+0000", 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10099335800000001 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.576265+0000", 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8279000000000001e-05 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677483+0000", 2024-09-16T04:42:15.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101218777 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677540+0000", 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6759999999999999e-05 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/32 ECSubWrite(tid=34, reqid=client.4728.0:211, at_version=38'19, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.486942+0000", 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.613400543000001, 2024-09-16T04:42:15.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19185692900000001, 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486942+0000", 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486942+0000", 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486944+0000", 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.697e-06 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.486979+0000", 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4623999999999999e-05 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.487008+0000", 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9166000000000002e-05 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.487028+0000", 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9749e-05 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.676989+0000", 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18996172 2024-09-16T04:42:15.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677016+0000", 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6804999999999998e-05 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.678706+0000", 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016896960000000001 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.678799+0000", 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3472000000000002e-05 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.11s4 44/41 ECSubWrite(tid=55, reqid=osd.4.0:55, at_version=44'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.213827+0000", 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.886515958, 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106368214, 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.144 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213827+0000", 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213827+0000", 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213828+0000", 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4000000000000002e-07 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213892+0000", 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4057e-05 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213919+0000", 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7512e-05 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.213944+0000", 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4399000000000001e-05 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224385+0000", 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010440974 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.224408+0000", 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3603000000000001e-05 2024-09-16T04:42:15.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.320130+0000", 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.095721732000000004 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.320195+0000", 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5396999999999997e-05 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/32 ECSubWrite(tid=62, reqid=osd.7.0:61, at_version=0'0, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.593284950000001, 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20295831, 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507061+0000", 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5059999999999998e-06 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507088+0000", 2024-09-16T04:42:15.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7764999999999999e-05 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507122+0000", 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3671999999999999e-05 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607262+0000", 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100139588 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707564+0000", 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100302577 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707595+0000", 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0482000000000001e-05 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.709942+0000", 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002347671 2024-09-16T04:42:15.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.710016+0000", 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4048999999999995e-05 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 48/47 ECSubWrite(tid=50, reqid=client.4728.0:308, at_version=48'23, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.741751829000002, 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19739056699999999, 2024-09-16T04:42:15.153 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358591+0000", 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358592+0000", 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1540000000000001e-06 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358618+0000", 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5622e-05 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358639+0000", 2024-09-16T04:42:15.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1288e-05 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.358654+0000", 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4278e-05 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360496+0000", 2024-09-16T04:42:15.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018424470000000001 2024-09-16T04:42:15.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360504+0000", 2024-09-16T04:42:15.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4100000000000008e-06 2024-09-16T04:42:15.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555946+0000", 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.195441898 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555982+0000", 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5469999999999998e-05 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 48/47 ECSubWrite(tid=51, reqid=client.4728.0:309, at_version=48'24, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.739962101, 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.196218006, 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360381+0000", 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360382+0000", 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.004e-06 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360399+0000", 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7278e-05 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360415+0000", 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6004000000000001e-05 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.360423+0000", 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1820000000000006e-06 2024-09-16T04:42:15.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555757+0000", 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19533410100000001 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.555766+0000", 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5590000000000006e-06 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.556544+0000", 2024-09-16T04:42:15.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00077832900000000002 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:58.556599+0000", 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4549000000000002e-05 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/37 ECSubWrite(tid=84, reqid=client.4728.0:380, at_version=49'15, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.895580733999999, 2024-09-16T04:42:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20242805, 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204766+0000", 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.523e-06 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205263+0000", 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00049713700000000003 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205312+0000", 2024-09-16T04:42:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8908e-05 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205338+0000", 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6602e-05 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207188+0000", 2024-09-16T04:42:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018498200000000001 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207215+0000", 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6322e-05 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407111+0000", 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.199896194 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407190+0000", 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9543999999999996e-05 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/37 ECSubWrite(tid=85, reqid=client.4728.0:381, at_version=49'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.893713258, 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20298417399999999, 2024-09-16T04:42:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206631+0000", 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206948+0000", 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031706300000000002 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206984+0000", 2024-09-16T04:42:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6256e-05 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207008+0000", 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3726e-05 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.405710+0000", 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19870283399999999 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.405729+0000", 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8116000000000001e-05 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.409550+0000", 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0038217580000000002 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.409614+0000", 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3621000000000006e-05 2024-09-16T04:42:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.173 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.173 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.173 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/37 ECSubWrite(tid=86, reqid=client.4728.0:382, at_version=49'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:15.173 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:00.208341+0000", 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.892002, 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20174012699999999, 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.208341+0000", 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.208341+0000", 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.208343+0000", 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9539999999999998e-06 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.208556+0000", 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000212938 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.208606+0000", 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0331000000000003e-05 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.308707+0000", 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100100358 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407280+0000", 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098573901000000005 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407303+0000", 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2314999999999999e-05 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.410013+0000", 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027104899999999999 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.410081+0000", 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7840000000000001e-05 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.177 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPushReply(3.11s0 55/53 [PushReplyOp(3:8f4ffdf1:::smithi16530038-24:head)])", 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:05.239227+0000", 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 9.8611158050000007, 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10719055399999999, 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239227+0000", 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239227+0000", 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239228+0000", 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239238+0000", 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6129999999999993e-06 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239262+0000", 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3626e-05 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239277+0000", 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4949999999999999e-05 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.346336+0000", 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.107059769 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.346418+0000", 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1403000000000006e-05 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPushReply(3.11s0 55/53 [PushReplyOp(3:8f4ffdf1:::smithi16530038-24:head)])", 2024-09-16T04:42:15.181 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:05.239658+0000", 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 9.8606847569999996, 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106460097, 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239658+0000", 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239658+0000", 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239659+0000", 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2699999999999999e-07 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239665+0000", 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4359999999999999e-06 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239683+0000", 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7825999999999998e-05 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.239694+0000", 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0383e-05 2024-09-16T04:42:15.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.345944+0000", 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106250557 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:05.346118+0000", 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017416799999999999 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 58/37 ECSubWrite(tid=116, reqid=client.4728.0:454, at_version=58'26, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.6963547639999996, 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23856791699999999, 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403990+0000", 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9769999999999999e-06 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404088+0000", 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7986999999999995e-05 2024-09-16T04:42:15.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404119+0000", 2024-09-16T04:42:15.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0975000000000001e-05 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404135+0000", 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5874000000000001e-05 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.540429+0000", 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13629391299999999 2024-09-16T04:42:15.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.540454+0000", 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5075999999999998e-05 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642480+0000", 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102026085 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642556+0000", 2024-09-16T04:42:15.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6030000000000002e-05 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 58/37 ECSubWrite(tid=117, reqid=client.4728.0:455, at_version=58'27, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.6961800289999998, 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23846093700000001, 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404164+0000", 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9299999999999998e-07 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404183+0000", 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9066e-05 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404196+0000", 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2999999999999999e-05 2024-09-16T04:42:15.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404205+0000", 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4229999999999994e-06 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.541363+0000", 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.137157952 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.541386+0000", 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3028999999999999e-05 2024-09-16T04:42:15.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642560+0000", 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101174403 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642624+0000", 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3471000000000003e-05 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 58/32 ECSubWrite(tid=116, reqid=client.4728.0:462, at_version=58'44, trim_to=0'0, roll_forward_to=57'42))", 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.2637351079999997, 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20377977999999999, 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836610+0000", 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836817+0000", 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020725999999999999 2024-09-16T04:42:15.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836849+0000", 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2149e-05 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.937028+0000", 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100178188 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.037317+0000", 2024-09-16T04:42:15.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100289105 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.037342+0000", 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5812e-05 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.040305+0000", 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002962059 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.040388+0000", 2024-09-16T04:42:15.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3153999999999997e-05 2024-09-16T04:42:15.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.200 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.200 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.200 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:15.201 INFO:teuthology.orchestra.run.smithi059.stderr:set 3.10 pg_upmap mapping to [2,7,1,14,5,3,15] 2024-09-16T04:42:15.202 INFO:tasks.rados.rados.0.smithi165.stdout:204: done (2 left) 2024-09-16T04:42:15.202 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 2 current snap is 15 2024-09-16T04:42:15.202 INFO:tasks.rados.rados.0.smithi165.stdout:203: expect (ObjNum 55 snap 1 seq_num 55) 2024-09-16T04:42:15.202 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:42:15.204 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:42:15.204 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:15.200+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:202: finishing copy_from racing read to smithi16530038-17 2024-09-16T04:42:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:202: finishing copy_from to smithi16530038-17 2024-09-16T04:42:15.224 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 21 (ObjNum 74 snap 5 seq_num 74) dirty exists 2024-09-16T04:42:15.224 INFO:tasks.rados.rados.0.smithi165.stdout:205: seq_num 88 ranges {32768=720896,753664=442368,1196032=327680} 2024-09-16T04:42:15.228 INFO:tasks.rados.rados.0.smithi165.stdout:205: writing smithi16530038-2 from 32768 to 753664 tid 1 2024-09-16T04:42:15.230 INFO:tasks.rados.rados.0.smithi165.stdout:205: writing smithi16530038-2 from 753664 to 1196032 tid 2 2024-09-16T04:42:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:205: writing smithi16530038-2 from 1196032 to 1523712 tid 3 2024-09-16T04:42:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:202: done (2 left) 2024-09-16T04:42:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:203: done (1 left) 2024-09-16T04:42:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:206: copy_from oid 29 from oid 40 current snap is 15 2024-09-16T04:42:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:207: rollback oid 38 current snap is 15 2024-09-16T04:42:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 38 to 13 2024-09-16T04:42:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:208: rmattr oid 8 current snap is 15 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:208: done (3 left) 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:209: rmattr oid 24 current snap is 15 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:209: done (3 left) 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:210: rmattr oid 4 current snap is 15 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:210: done (3 left) 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:211: read oid 1 snap -1 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:211: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:212: read oid 6 snap -1 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:212: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:42:15.234 INFO:tasks.rados.rados.0.smithi165.stdout:213: setattr oid 36 current snap is 15 2024-09-16T04:42:15.235 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 20 current snap is 15 2024-09-16T04:42:15.235 INFO:tasks.rados.rados.0.smithi165.stdout:214: seq_num 89 ranges {180224=737280,917504=573440,1490944=196608} 2024-09-16T04:42:15.248 INFO:tasks.rados.rados.0.smithi165.stdout:214: writing smithi16530038-20 from 180224 to 917504 tid 1 2024-09-16T04:42:15.255 INFO:tasks.rados.rados.0.smithi165.stdout:214: writing smithi16530038-20 from 917504 to 1490944 tid 2 2024-09-16T04:42:15.258 INFO:tasks.rados.rados.0.smithi165.stdout:214: writing smithi16530038-20 from 1490944 to 1687552 tid 3 2024-09-16T04:42:15.258 INFO:tasks.rados.rados.0.smithi165.stdout:215: copy_from oid 34 from oid 23 current snap is 15 2024-09-16T04:42:15.258 INFO:tasks.rados.rados.0.smithi165.stdout:216: snap_create 2024-09-16T04:42:15.303 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:15.295+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:15.378 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.359168+0000", 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.018665702999999999, 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018705672, 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.379 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359168+0000", 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359168+0000", 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359170+0000", 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.73e-06 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359345+0000", 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017533199999999999 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359376+0000", 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1319999999999998e-05 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359393+0000", 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6399e-05 2024-09-16T04:42:15.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359481+0000", 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8410000000000002e-05 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359494+0000", 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2357999999999999e-05 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.359428+0000", 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.018405950000000001, 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018550520000000001, 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359428+0000", 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.383 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359428+0000", 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359429+0000", 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0399999999999995e-07 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359482+0000", 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3315e-05 2024-09-16T04:42:15.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359495+0000", 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3386e-05 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359505+0000", 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.73e-06 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.360068+0000", 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00056262700000000001 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.360077+0000", 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3929999999999997e-06 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.359524+0000", 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.018310069000000002, 2024-09-16T04:42:15.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018547559000000002, 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359524+0000", 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359524+0000", 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.387 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359524+0000", 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0299999999999998e-07 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359542+0000", 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7972999999999999e-05 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.388 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359553+0000", 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.08e-05 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.359562+0000", 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9360000000000005e-06 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 3 2024-09-16T04:42:15.389 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:15.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:42:15.404 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:15.399+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:15.441 INFO:tasks.rados.rados.0.smithi165.stdout:212: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:42:15.505 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:15.499+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:15.511 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:15.511 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:15.511 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:15.511 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:15.512 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:15.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:42:15.606 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:15.604+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:15.644 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 38 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:15.646 INFO:tasks.rados.rados.0.smithi165.stdout:207: finishing rollback tid 0 to smithi16530038-38 2024-09-16T04:42:15.651 INFO:tasks.rados.rados.0.smithi165.stdout:215: finishing copy_from to smithi16530038-34 2024-09-16T04:42:15.651 INFO:tasks.rados.rados.0.smithi165.stdout:215: got expected ENOENT (src dne) 2024-09-16T04:42:15.651 INFO:tasks.rados.rados.0.smithi165.stdout:214: finishing write tid 1 to smithi16530038-20 2024-09-16T04:42:15.651 INFO:tasks.rados.rados.0.smithi165.stdout:215: finishing copy_from racing read to smithi16530038-34 2024-09-16T04:42:15.651 INFO:tasks.rados.rados.0.smithi165.stdout:214: finishing write tid 2 to smithi16530038-20 2024-09-16T04:42:15.652 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:15.652 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:15.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:15.652 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:15.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.652 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:156 3.9s0 3:9de467d5:::smithi16530038-27:head [append 0~475136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.383392+0000", 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.265329453, 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40721533599999998, 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 156 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.653 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383392+0000", 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383392+0000", 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383395+0000", 2024-09-16T04:42:15.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5560000000000002e-06 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383797+0000", 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040218300000000002 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.383823+0000", 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6216000000000001e-05 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.483957+0000", 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10013338300000001 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484413+0000", 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00045616999999999999 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484595+0000", 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000182566 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.485946+0000", 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001351196 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589202+0000", 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10325537 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790481+0000", 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20127900600000001 2024-09-16T04:42:15.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790608+0000", 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012668999999999999 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:157 3.9s0 3:9de467d5:::smithi16530038-27:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.164714128, 2024-09-16T04:42:15.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51218579099999995, 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 157 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484010+0000", 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0329999999999998e-06 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484272+0000", 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026256800000000002 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484296+0000", 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3509000000000001e-05 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484308+0000", 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2699e-05 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584144+0000", 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099835667000000003 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584304+0000", 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016037700000000001 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.585781+0000", 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001476455 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589205+0000", 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0034239499999999998 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996077+0000", 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40687225300000002 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996193+0000", 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011628 2024-09-16T04:42:15.663 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:158 3.9s0 3:9de467d5:::smithi16530038-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.164390600000001, 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51808159200000004, 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.664 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 158 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484332+0000", 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484339+0000", 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8630000000000002e-06 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484355+0000", 2024-09-16T04:42:15.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6473e-05 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484364+0000", 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8950000000000004e-06 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.586917+0000", 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102553035 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.587177+0000", 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026005499999999998 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.588403+0000", 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001225152 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589911+0000", 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015086260000000001 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002165+0000", 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41225355800000002 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002413+0000", 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024783200000000002 2024-09-16T04:42:15.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.9s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.164321790999999, 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.62506688499999996, 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.670 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484401+0000", 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484405+0000", 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.827e-06 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484416+0000", 2024-09-16T04:42:15.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1609e-05 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484425+0000", 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790928+0000", 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30650327300000002 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:15.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.791067+0000", 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013860200000000001 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003346+0000", 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212279577 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003611+0000", 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026456899999999998 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109440+0000", 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105828857 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109467+0000", 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6900999999999999e-05 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.675 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 36/32 ECSubWriteReply(tid=19, last_complete=36'4, committed=1, applied=1))", 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.499445+0000", 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.149276319999998, 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28634094100000002, 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499445+0000", 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499445+0000", 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499447+0000", 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2529999999999999e-06 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499453+0000", 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4570000000000002e-06 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499483+0000", 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9850999999999999e-05 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.499515+0000", 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2064999999999999e-05 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.785726+0000", 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.286210878 2024-09-16T04:42:15.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.785786+0000", 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0436999999999998e-05 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.19s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.541369184000001, 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.976424197, 2024-09-16T04:42:15.679 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107354+0000", 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9829999999999999e-06 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107362+0000", 2024-09-16T04:42:15.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6129999999999998e-06 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107388+0000", 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5675000000000001e-05 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207482+0000", 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009412299999999 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207644+0000", 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000161741 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207649+0000", 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0359999999999997e-06 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082558+0000", 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8749096240000001 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082784+0000", 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022614900000000001 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.083777+0000", 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099225300000000002 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:178 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.570607758000001, 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1069151189999999, 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 178 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078116+0000", 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078127+0000", 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.062e-05 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078151+0000", 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4304999999999999e-05 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078175+0000", 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4006e-05 2024-09-16T04:42:15.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776496+0000", 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69832100500000005 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776503+0000", 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.533e-06 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185014+0000", 2024-09-16T04:42:15.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40851148700000001 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185029+0000", 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4780000000000001e-05 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:179 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.570512396000002, 2024-09-16T04:42:15.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1068670979999999, 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 179 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078211+0000", 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.44e-06 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078220+0000", 2024-09-16T04:42:15.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078244+0000", 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3867999999999999e-05 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078260+0000", 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6203000000000001e-05 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780436+0000", 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70217526500000005 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780441+0000", 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4620000000000004e-06 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185067+0000", 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40462590799999998 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185076+0000", 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3219999999999999e-06 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:180 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.57043934, 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.106828843, 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.694 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 180 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078284+0000", 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078291+0000", 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.657e-06 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078310+0000", 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078325+0000", 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4772e-05 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981190+0000", 2024-09-16T04:42:15.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90286589500000003 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981194+0000", 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0790000000000002e-06 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185100+0000", 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20390586399999999 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185111+0000", 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0794e-05 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:185 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.550123361000001, 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98955797199999995, 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.699 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098600+0000", 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098611+0000", 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0423e-05 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098642+0000", 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1532000000000002e-05 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098658+0000", 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5996000000000001e-05 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683726+0000", 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58506727400000003 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683731+0000", 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088142+0000", 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40441113000000001 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088156+0000", 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4148000000000001e-05 2024-09-16T04:42:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:186 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.550035306000002, 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.089963279, 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 186 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098687+0000", 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.063e-06 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098693+0000", 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6230000000000002e-06 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098705+0000", 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2019e-05 2024-09-16T04:42:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198804+0000", 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098513 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683799+0000", 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48499579100000001 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683801+0000", 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0889999999999998e-06 2024-09-16T04:42:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188641+0000", 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50483997300000005 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188650+0000", 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.208e-06 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:187 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.449885902999998, 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98988477799999997, 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 187 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198838+0000", 2024-09-16T04:42:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.86e-06 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198846+0000", 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0059999999999992e-06 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198864+0000", 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8468999999999999e-05 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198876+0000", 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1443e-05 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783957+0000", 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58508126900000001 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783962+0000", 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7230000000000001e-06 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188710+0000", 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40474714899999997 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188721+0000", 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0859000000000001e-05 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s4 38/37 ECSubWrite(tid=5, reqid=client.4728.0:189, at_version=38'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.701042+0000", 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.947679260000001, 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38743471499999999, 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701042+0000", 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701042+0000", 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701044+0000", 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.03e-06 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701364+0000", 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032004700000000001 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701407+0000", 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2710000000000003e-05 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.701433+0000", 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5409e-05 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.983563+0000", 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.282130618 2024-09-16T04:42:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.983572+0000", 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088375+0000", 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.104802627 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088477+0000", 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010224499999999999 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s4 42/37 ECSubWrite(tid=9, reqid=client.4728.0:269, at_version=42'6, trim_to=0'0, roll_forward_to=38'4))", 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.521672+0000", 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 23.127049459999999, 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29629296900000002, 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.521672+0000", 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.521672+0000", 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.521674+0000", 2024-09-16T04:42:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.477e-06 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.522396+0000", 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00072250799999999996 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.522422+0000", 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5505999999999999e-05 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.522436+0000", 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4435e-05 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.616799+0000", 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094362980999999999 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.616812+0000", 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3239e-05 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817885+0000", 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.201072954 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817965+0000", 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9869000000000004e-05 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 42/41 ECSubWriteReply(tid=33, last_complete=42'8, committed=1, applied=1))", 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.610676+0000", 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 23.038045428, 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20704228299999999, 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610676+0000", 2024-09-16T04:42:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610676+0000", 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610677+0000", 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610680+0000", 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0699999999999998e-06 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610693+0000", 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3162e-05 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610700+0000", 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2969999999999998e-06 2024-09-16T04:42:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817685+0000", 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206984848 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817718+0000", 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3315999999999997e-05 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 42/41 ECSubWriteReply(tid=29, last_complete=42'6, committed=1, applied=1))", 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.610878+0000", 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 23.037843177999999, 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.207101543, 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610878+0000", 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610878+0000", 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610879+0000", 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4600000000000003e-07 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610884+0000", 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2139999999999998e-06 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610903+0000", 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.610914+0000", 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1392999999999999e-05 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817821+0000", 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206906426 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.817980+0000", 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000159419 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 44/41 ECSubWriteReply(tid=43, last_complete=44'19, committed=1, applied=1))", 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.614971+0000", 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.033750223999998, 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21339023400000001, 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614971+0000", 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614971+0000", 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614974+0000", 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.12e-06 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.614985+0000", 2024-09-16T04:42:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1293e-05 2024-09-16T04:42:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615025+0000", 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0500999999999997e-05 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615047+0000", 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1239000000000001e-05 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828311+0000", 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.213264333 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828362+0000", 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0748000000000003e-05 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 44/41 ECSubWriteReply(tid=44, last_complete=44'20, committed=1, applied=1))", 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.615085+0000", 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.033636372, 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21334994199999999, 2024-09-16T04:42:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615085+0000", 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615085+0000", 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615086+0000", 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.009e-06 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615092+0000", 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6529999999999999e-06 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615116+0000", 2024-09-16T04:42:15.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3632999999999999e-05 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.615130+0000", 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4759e-05 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828408+0000", 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.213277732 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.828435+0000", 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7155999999999999e-05 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.19s0 44/41 ECSubWriteReply(tid=43, last_complete=44'19, committed=1, applied=1))", 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.726134+0000", 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.922587904, 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.303476629, 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.742 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726134+0000", 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726134+0000", 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726135+0000", 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726166+0000", 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1001999999999999e-05 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726182+0000", 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6846000000000001e-05 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.726193+0000", 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1129e-05 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:55.029574+0000", 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30338042300000001 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:55.029610+0000", 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6439000000000001e-05 2024-09-16T04:42:15.745 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 49/47 ECSubWriteReply(tid=54, last_complete=48'26, committed=1, applied=1))", 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:59.275145+0000", 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.373576296, 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30028958100000003, 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.275145+0000", 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.275145+0000", 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.275146+0000", 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0200000000000001e-07 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.275151+0000", 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5260000000000004e-06 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.275159+0000", 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5169999999999999e-06 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.375233+0000", 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10007389899999999 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.575399+0000", 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20016598199999999 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.575435+0000", 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5855000000000001e-05 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:15.749 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:15.750 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:15.751 INFO:tasks.rados.rados.0.smithi165.stdout:214: finishing write tid 3 to smithi16530038-20 2024-09-16T04:42:15.751 INFO:tasks.rados.rados.0.smithi165.stdout:211: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:15.752 INFO:tasks.rados.rados.0.smithi165.stdout:214: finishing write tid 4 to smithi16530038-20 2024-09-16T04:42:15.752 INFO:tasks.rados.rados.0.smithi165.stdout:207: finishing rollback tid 1 to smithi16530038-38 2024-09-16T04:42:15.752 INFO:tasks.rados.rados.0.smithi165.stdout:207: finishing rollback tid 2 to smithi16530038-38 2024-09-16T04:42:15.752 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 25 (ObjNum 67 snap 4 seq_num 67) dirty exists 2024-09-16T04:42:15.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:42:15.753 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:15.748+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:15.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:15.839 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 1 to smithi16530038-2 2024-09-16T04:42:15.840 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 2 to smithi16530038-2 2024-09-16T04:42:15.840 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 3 to smithi16530038-2 2024-09-16T04:42:15.840 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 4 to smithi16530038-2 2024-09-16T04:42:15.852 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:15.847+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:15.884 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:15.884 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:15.884 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:15.884 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:15.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:42:15.921 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:15.943 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:15.943 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:15.952 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:15.947+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:15.965 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:15.965 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:15.987 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:16.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:16.005 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:16.017 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:16.018 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:16.026 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:16.027 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:16.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:42:16.043 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 5 to smithi16530038-2 2024-09-16T04:42:16.043 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 2 updating version 0 to 105 2024-09-16T04:42:16.043 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 2 updating version 105 to 108 2024-09-16T04:42:16.043 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 2 version 108 is already newer than 106 2024-09-16T04:42:16.043 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 2 version 108 is already newer than 107 2024-09-16T04:42:16.043 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 108 (ObjNum 88 snap 15 seq_num 88) dirty exists 2024-09-16T04:42:16.043 INFO:tasks.rados.rados.0.smithi165.stdout:205: left oid 2 (ObjNum 88 snap 15 seq_num 88) 2024-09-16T04:42:16.048 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:16.053 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:16.051+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:16.068 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:16.068 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:16.069 INFO:tasks.rados.rados.0.smithi165.stdout:206: finishing copy_from racing read to smithi16530038-29 2024-09-16T04:42:16.069 INFO:tasks.rados.rados.0.smithi165.stdout:214: finishing write tid 5 to smithi16530038-20 2024-09-16T04:42:16.069 INFO:tasks.rados.rados.0.smithi165.stdout:214: oid 20 updating version 0 to 46 2024-09-16T04:42:16.069 INFO:tasks.rados.rados.0.smithi165.stdout:214: oid 20 updating version 46 to 48 2024-09-16T04:42:16.069 INFO:tasks.rados.rados.0.smithi165.stdout:214: oid 20 version 48 is already newer than 45 2024-09-16T04:42:16.070 INFO:tasks.rados.rados.0.smithi165.stdout:214: oid 20 version 48 is already newer than 47 2024-09-16T04:42:16.070 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 48 (ObjNum 89 snap 15 seq_num 89) dirty exists 2024-09-16T04:42:16.070 INFO:tasks.rados.rados.0.smithi165.stdout:214: left oid 20 (ObjNum 89 snap 15 seq_num 89) 2024-09-16T04:42:16.088 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:16.110 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:16.110 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:16.134 INFO:tasks.rados.rados.0.smithi165.stdout:205: done (8 left) 2024-09-16T04:42:16.134 INFO:tasks.rados.rados.0.smithi165.stdout:207: done (7 left) 2024-09-16T04:42:16.135 INFO:tasks.rados.rados.0.smithi165.stdout:211: done (6 left) 2024-09-16T04:42:16.135 INFO:tasks.rados.rados.0.smithi165.stdout:212: done (5 left) 2024-09-16T04:42:16.135 INFO:tasks.rados.rados.0.smithi165.stdout:213: done (4 left) 2024-09-16T04:42:16.135 INFO:tasks.rados.rados.0.smithi165.stdout:214: done (3 left) 2024-09-16T04:42:16.135 INFO:tasks.rados.rados.0.smithi165.stdout:215: done (2 left) 2024-09-16T04:42:16.135 INFO:tasks.rados.rados.0.smithi165.stdout:216: done (1 left) 2024-09-16T04:42:16.135 INFO:tasks.rados.rados.0.smithi165.stdout:217: delete oid 36 current snap is 16 2024-09-16T04:42:16.138 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:16.138 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:16.154 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:16.151+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:16.159 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:16.159 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:16.168 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:16.168 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:16.176 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:16.177 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:16.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:42:16.200 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:16.200 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:16.223 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:16.223 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:16.255 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:16.247+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:16.263 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:16.348 INFO:tasks.rados.rados.0.smithi165.stdout:217: done (1 left) 2024-09-16T04:42:16.348 INFO:tasks.rados.rados.0.smithi165.stdout:218: delete oid 4 current snap is 16 2024-09-16T04:42:16.352 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:16.352 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:16.355 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:16.351+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:16.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:16.456 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:16.451+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:16.470 INFO:tasks.rados.rados.0.smithi165.stdout:206: finishing copy_from to smithi16530038-29 2024-09-16T04:42:16.470 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T04:42:16.487 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:16.487 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:16.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:16.558 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:16.555+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:16.570 INFO:tasks.rados.rados.0.smithi165.stdout:206: done (1 left) 2024-09-16T04:42:16.570 INFO:tasks.rados.rados.0.smithi165.stdout:218: done (0 left) 2024-09-16T04:42:16.571 INFO:tasks.rados.rados.0.smithi165.stdout:219: copy_from oid 20 from oid 42 current snap is 16 2024-09-16T04:42:16.571 INFO:tasks.rados.rados.0.smithi165.stdout:220: delete oid 5 current snap is 16 2024-09-16T04:42:16.621 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:16.621 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:16.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:16.682 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:16.679+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:16.756 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:16.763 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:16.759+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:16.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:16.778 INFO:tasks.rados.rados.0.smithi165.stdout:220: done (1 left) 2024-09-16T04:42:16.778 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 13 current snap is 16 2024-09-16T04:42:16.778 INFO:tasks.rados.rados.0.smithi165.stdout:221: seq_num 93 ranges {1671168=802816,2473984=638976,3112960=131072} 2024-09-16T04:42:16.798 INFO:tasks.rados.rados.0.smithi165.stdout:221: writing smithi16530038-13 from 1671168 to 2473984 tid 1 2024-09-16T04:42:16.803 INFO:tasks.rados.rados.0.smithi165.stdout:221: writing smithi16530038-13 from 2473984 to 3112960 tid 2 2024-09-16T04:42:16.805 INFO:tasks.rados.rados.0.smithi165.stdout:221: writing smithi16530038-13 from 3112960 to 3244032 tid 3 2024-09-16T04:42:16.805 INFO:tasks.rados.rados.0.smithi165.stdout:222: read oid 19 snap -1 2024-09-16T04:42:16.805 INFO:tasks.rados.rados.0.smithi165.stdout:222: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T04:42:16.805 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 21 current snap is 16 2024-09-16T04:42:16.805 INFO:tasks.rados.rados.0.smithi165.stdout:219: finishing copy_from racing read to smithi16530038-20 2024-09-16T04:42:16.805 INFO:tasks.rados.rados.0.smithi165.stdout:223: seq_num 94 ranges {393216=425984,819200=786432,1605632=376832} 2024-09-16T04:42:16.811 INFO:tasks.rados.rados.0.smithi165.stdout:223: writing smithi16530038-21 from 393216 to 819200 tid 1 2024-09-16T04:42:16.818 INFO:tasks.rados.rados.0.smithi165.stdout:223: writing smithi16530038-21 from 819200 to 1605632 tid 2 2024-09-16T04:42:16.821 INFO:tasks.rados.rados.0.smithi165.stdout:223: writing smithi16530038-21 from 1605632 to 1982464 tid 3 2024-09-16T04:42:16.821 INFO:tasks.rados.rados.0.smithi165.stdout:224: delete oid 22 current snap is 16 2024-09-16T04:42:16.864 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:16.859+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:16.887 INFO:tasks.rados.rados.0.smithi165.stdout:219: finishing copy_from to smithi16530038-20 2024-09-16T04:42:16.888 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 49 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:42:16.900 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:16.901 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:16.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:16.965 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:16.959+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:17.016 INFO:tasks.rados.rados.0.smithi165.stdout:222: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T04:42:17.066 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:17.064+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 1 to smithi16530038-21 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:219: done (4 left) 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:222: done (3 left) 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:224: done (2 left) 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:225: read oid 9 snap -1 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:225: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:226: copy_from oid 4 from oid 7 current snap is 16 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:227: copy_from oid 33 from oid 34 current snap is 16 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:228: read oid 27 snap 11 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:228: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-16T04:42:17.070 INFO:tasks.rados.rados.0.smithi165.stdout:229: delete oid 17 current snap is 16 2024-09-16T04:42:17.081 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:17.081 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:17.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:42:17.167 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:17.164+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:17.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:42:17.212 INFO:tasks.rados.rados.0.smithi165.stdout:221: finishing write tid 1 to smithi16530038-13 2024-09-16T04:42:17.213 INFO:tasks.rados.rados.0.smithi165.stdout:221: finishing write tid 2 to smithi16530038-13 2024-09-16T04:42:17.213 INFO:tasks.rados.rados.0.smithi165.stdout:221: finishing write tid 3 to smithi16530038-13 2024-09-16T04:42:17.214 INFO:tasks.rados.rados.0.smithi165.stdout:221: finishing write tid 4 to smithi16530038-13 2024-09-16T04:42:17.220 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:17.220 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:17.229 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 2 to smithi16530038-21 2024-09-16T04:42:17.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:42:17.267 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:17.263+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:17.272 INFO:tasks.rados.rados.0.smithi165.stdout:227: finishing copy_from to smithi16530038-33 2024-09-16T04:42:17.273 INFO:tasks.rados.rados.0.smithi165.stdout:227: got expected ENOENT (src dne) 2024-09-16T04:42:17.283 INFO:tasks.rados.rados.0.smithi165.stdout:227: finishing copy_from racing read to smithi16530038-33 2024-09-16T04:42:17.330 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 3 to smithi16530038-21 2024-09-16T04:42:17.362 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:17.362 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:17.369 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:17.363+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:17.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:42:17.470 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:17.467+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:17.532 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 4 to smithi16530038-21 2024-09-16T04:42:17.547 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:17.547 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:17.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:226: finishing copy_from racing read to smithi16530038-4 2024-09-16T04:42:17.593 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:17.587+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:17.595 INFO:tasks.rados.rados.0.smithi165.stdout:228: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-16T04:42:17.674 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:17.672+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:17.692 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:17.692 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:17.706 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:17.775 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:17.771+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:17.794 INFO:tasks.rados.rados.0.smithi165.stdout:226: finishing copy_from to smithi16530038-4 2024-09-16T04:42:17.794 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:17.798 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 5 to smithi16530038-21 2024-09-16T04:42:17.798 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 21 updating version 0 to 18 2024-09-16T04:42:17.798 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 21 updating version 18 to 19 2024-09-16T04:42:17.798 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 21 version 19 is already newer than 17 2024-09-16T04:42:17.798 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 21 version 19 is already newer than 16 2024-09-16T04:42:17.799 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 19 (ObjNum 94 snap 16 seq_num 94) dirty exists 2024-09-16T04:42:17.799 INFO:tasks.rados.rados.0.smithi165.stdout:223: left oid 21 (ObjNum 94 snap 16 seq_num 94) 2024-09-16T04:42:17.833 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:17.833 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:17.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:17.889 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:17.887+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:17.977 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:17.971+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:18.018 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:18.018 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:18.032 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:18.078 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:18.075+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:18.159 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:18.159 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:18.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:18.178 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:18.175+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:18.279 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:18.275+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:42:18.286 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:18.305 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:18.305 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:18.324 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:18.398 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:18.396+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:18.443 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:18.444 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:18.465 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:18.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:18.499 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:18.496+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:18.511 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:18.511 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:18.531 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:18.531 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:137 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:18.534 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.470532083000002, 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50711364199999998, 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 137 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:18.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062314+0000", 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062318+0000", 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1710000000000004e-06 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062327+0000", 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0089999999999996e-06 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162444+0000", 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011658 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163173+0000", 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072917800000000003 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163440+0000", 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026697899999999998 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164004+0000", 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056357499999999997 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164775+0000", 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00077136499999999998 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569249+0000", 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.404473996 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569427+0000", 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177906 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:138 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.3703632, 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.71018989099999996, 2024-09-16T04:42:18.540 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 138 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162484+0000", 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162491+0000", 2024-09-16T04:42:18.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162507+0000", 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5141e-05 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162514+0000", 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8460000000000008e-06 2024-09-16T04:42:18.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165370+0000", 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028555719999999998 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165438+0000", 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8466999999999993e-05 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570190+0000", 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40475178899999997 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570271+0000", 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0437000000000003e-05 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.872672+0000", 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30240151100000001 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.545 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:181 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.440489933999999, 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.514197601, 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 181 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092358+0000", 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9500000000000001e-06 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093295+0000", 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00093678900000000005 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093330+0000", 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5039000000000001e-05 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093348+0000", 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7210999999999998e-05 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269509+0000", 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17616195900000001 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269804+0000", 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029419599999999999 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272744+0000", 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029401789999999998 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279866+0000", 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007122499 2024-09-16T04:42:18.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606426+0000", 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32655929 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606553+0000", 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000127489 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:182 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.434161489000001, 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50813697599999996, 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 182 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.552 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0260000000000002e-06 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099161+0000", 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000474556 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099192+0000", 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1612000000000002e-05 2024-09-16T04:42:18.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099207+0000", 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4796e-05 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275225+0000", 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17601765799999999 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275471+0000", 2024-09-16T04:42:18.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024600099999999998 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001437422 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280012+0000", 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031043899999999998 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606726+0000", 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32671404399999998 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606821+0000", 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4470999999999997e-05 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:183 3.bs0 3:d388eed4:::smithi16530038-28:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.557 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.433611079999999, 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60643071599999998, 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 183 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099235+0000", 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.017e-06 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099242+0000", 2024-09-16T04:42:18.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4230000000000002e-06 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099256+0000", 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3936e-05 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099264+0000", 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278076+0000", 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17881208600000001 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278318+0000", 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024238100000000001 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279368+0000", 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010492209999999999 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281553+0000", 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021853739999999999 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705572+0000", 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.42401889100000001 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705665+0000", 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3173999999999999e-05 2024-09-16T04:42:18.562 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:191 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.431750924999999, 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68460226999999996, 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 191 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:18.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101096+0000", 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-06 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101104+0000", 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1240000000000005e-06 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101122+0000", 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8045000000000001e-05 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101130+0000", 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2670000000000006e-06 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280859+0000", 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179728363 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280864+0000", 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7370000000000003e-06 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.405994+0000", 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.12512986500000001 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.406243+0000", 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024852700000000001 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.785697+0000", 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.37945393900000002 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:192 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.431698706999999, 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68518017200000003, 2024-09-16T04:42:18.568 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 192 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101154+0000", 2024-09-16T04:42:18.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3300000000000004e-06 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101160+0000", 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.348e-06 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101164+0000", 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2899999999999996e-06 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281071+0000", 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179906749 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281074+0000", 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.976e-06 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506776+0000", 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22570168299999999 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506893+0000", 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011752300000000001 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786327+0000", 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27943377000000003 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:199 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.573 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.431624348, 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50411928699999997, 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 199 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101221+0000", 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101222+0000", 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3799999999999997e-07 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101227+0000", 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8559999999999998e-06 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101240+0000", 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3485e-05 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101248+0000", 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3699999999999995e-06 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280982+0000", 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17973399200000001 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281111+0000", 2024-09-16T04:42:18.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012887099999999999 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.605340+0000", 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32422917499999998 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:200 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:18.578 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.431587474000001, 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50471661899999998, 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 200 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:18.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101258+0000", 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7300000000000001e-07 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101262+0000", 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.14e-06 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101269+0000", 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6399999999999997e-06 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101275+0000", 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6910000000000002e-06 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281623+0000", 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18034850899999999 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281792+0000", 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016865700000000001 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.605975+0000", 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32418250900000001 2024-09-16T04:42:18.582 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:201 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.431565620000001, 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68324074999999995, 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.583 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 201 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101282+0000", 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9750000000000001e-06 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101287+0000", 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.651e-06 2024-09-16T04:42:18.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.201419+0000", 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10013264099999999 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482351+0000", 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28093166899999999 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482482+0000", 2024-09-16T04:42:18.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013094199999999999 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.784521+0000", 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30203853899999999 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:354 3.17s0 3:ed82f61a:::smithi16530038-1:3 [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:58.147804+0000", 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.385041566999998, 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.41067514300000002, 2024-09-16T04:42:18.587 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 354 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147804+0000", 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147804+0000", 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147805+0000", 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5e-06 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147817+0000", 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1253000000000001e-05 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147843+0000", 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6055999999999999e-05 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.147858+0000", 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4949999999999999e-05 2024-09-16T04:42:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.148045+0000", 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000187837 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.148055+0000", 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1670000000000005e-06 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.156045+0000", 2024-09-16T04:42:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079905459999999994 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.156173+0000", 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012744300000000001 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.558479+0000", 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40230639099999999 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:372 3.1s0 3:8583b9f2:::smithi16530038-39:3 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.351563322000001, 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.58438020099999999, 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 372 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181283+0000", 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1100000000000005e-07 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181288+0000", 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4739999999999996e-06 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181300+0000", 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1266e-05 2024-09-16T04:42:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181308+0000", 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5669999999999995e-06 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181619+0000", 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031025600000000002 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181627+0000", 2024-09-16T04:42:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3329999999999997e-06 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288322+0000", 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669511499999999 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288524+0000", 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202511 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.765662+0000", 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.47713786800000002 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:452 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.273891466, 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57389822599999996, 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 452 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258957+0000", 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9510000000000002e-06 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260061+0000", 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011037410000000001 2024-09-16T04:42:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260099+0000", 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8401e-05 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260123+0000", 2024-09-16T04:42:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3808999999999999e-05 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260226+0000", 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010314000000000001 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260310+0000", 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3652000000000005e-05 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.261217+0000", 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090691199999999995 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.262686+0000", 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014689 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832721+0000", 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57003547600000004 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832852+0000", 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013124400000000001 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:453 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~622592] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.263213592, 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56351327799999995, 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.607 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 453 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269634+0000", 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2069999999999998e-06 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270168+0000", 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053375500000000002 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270185+0000", 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7231000000000002e-05 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270194+0000", 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4349999999999997e-06 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270266+0000", 2024-09-16T04:42:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2702999999999998e-05 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270366+0000", 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9485e-05 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271233+0000", 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00086726900000000005 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374033+0000", 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10280007300000001 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833038+0000", 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45900526899999999 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833145+0000", 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106851 2024-09-16T04:42:18.612 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:454 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~147456] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.260948812000001, 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56224230100000006, 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 454 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271898+0000", 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.364e-06 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272075+0000", 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017664100000000001 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272088+0000", 2024-09-16T04:42:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3192e-05 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272097+0000", 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4720000000000001e-06 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372245+0000", 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10014781 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372332+0000", 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7258999999999999e-05 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372854+0000", 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052174600000000002 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374728+0000", 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018736219999999999 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834034+0000", 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45930626099999999 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834139+0000", 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000104934 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.618 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:455 3.1s0 3:803446a7:::smithi16530038-31:head [setxattr _header (57) in=7b] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.260724449, 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.66155386500000002, 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 455 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272122+0000", 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:42:18.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272127+0000", 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5369999999999997e-06 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272135+0000", 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272141+0000", 2024-09-16T04:42:18.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5999999999999997e-06 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373376+0000", 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101234779 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373445+0000", 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9507999999999999e-05 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373742+0000", 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029695500000000001 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374731+0000", 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098912400000000008 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933530+0000", 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.55879828600000003 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933675+0000", 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145353 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:456 3.1s0 3:803446a7:::smithi16530038-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.260693266000001, 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.76406687699999998, 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.624 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 456 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:18.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272153+0000", 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0100000000000005e-07 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272155+0000", 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.638e-06 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272165+0000", 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1579999999999994e-06 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272171+0000", 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373963+0000", 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10179178799999999 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374024+0000", 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0791999999999997e-05 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934462+0000", 2024-09-16T04:42:18.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56043857600000002 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934706+0000", 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024387000000000001 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.036219+0000", 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10151300000000001 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.629 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:457 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~655360] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.297257+0000", 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.235588763999999, 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.412841456, 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 457 2024-09-16T04:42:18.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297257+0000", 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297257+0000", 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297259+0000", 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.979e-06 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297936+0000", 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00067768099999999996 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297958+0000", 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1838e-05 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.297969+0000", 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0669000000000001e-05 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.298054+0000", 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5420999999999994e-05 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.298187+0000", 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013230499999999999 2024-09-16T04:42:18.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011698640000000001 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.300800+0000", 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014438459999999999 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.709969+0000", 2024-09-16T04:42:18.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40916917000000003 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.710098+0000", 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012868299999999999 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:458 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.229424202000001, 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50818626099999997, 2024-09-16T04:42:18.635 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 458 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.636 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303426+0000", 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5870000000000003e-06 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303961+0000", 2024-09-16T04:42:18.637 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053495599999999995 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303975+0000", 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3835999999999999e-05 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404043+0000", 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100068845 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404113+0000", 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9350999999999994e-05 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404242+0000", 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012877400000000001 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405312+0000", 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001070812 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407819+0000", 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002506131 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811485+0000", 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40366653800000002 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811608+0000", 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012243099999999999 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.640 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:461 3.bs0 3:d1897791:::smithi16530038-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.1286915, 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.61182177000000004, 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:18.641 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 461 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.642 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6399999999999998e-07 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404155+0000", 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2550000000000001e-06 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404160+0000", 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0890000000000002e-06 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404162+0000", 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8399999999999999e-06 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507951+0000", 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103788373 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.508127+0000", 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017614900000000001 2024-09-16T04:42:18.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813061+0000", 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30493453100000001 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813333+0000", 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027161699999999998 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.015976+0000", 2024-09-16T04:42:18.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20264265200000001 2024-09-16T04:42:18.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.646 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.646 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:18.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:18.648 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:18.649 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:18.649 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:18.650 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:18.651 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:18.651 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:18.643+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:18.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:18.668 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:18.693 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:18.715 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:18.738 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:18.738 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:18.748 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:18.743+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:18.761 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:18.785 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:18.785 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:18.801 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:18.801 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:18.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.801 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 68/67 ECSubWrite(tid=72, reqid=client.4728.0:567, at_version=68'49, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:18.801 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:18.704856+0000", 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.095143670999999999, 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.09516935, 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:18.704856+0000", 2024-09-16T04:42:18.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:18.704856+0000", 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:18.704856+0000", 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:42:18.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:18.704881+0000", 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4615000000000001e-05 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:18.704896+0000", 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5077e-05 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:18.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:18.704907+0000", 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0643e-05 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 1 2024-09-16T04:42:18.805 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:18.817 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:42:18.862 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:18.859+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:18.901 INFO:tasks.rados.rados.0.smithi165.stdout:223: done (6 left) 2024-09-16T04:42:18.901 INFO:tasks.rados.rados.0.smithi165.stdout:226: done (5 left) 2024-09-16T04:42:18.901 INFO:tasks.rados.rados.0.smithi165.stdout:227: done (4 left) 2024-09-16T04:42:18.901 INFO:tasks.rados.rados.0.smithi165.stdout:228: done (3 left) 2024-09-16T04:42:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:229: done (2 left) 2024-09-16T04:42:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:230: rollback oid 42 current snap is 16 2024-09-16T04:42:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 15 2024-09-16T04:42:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:231: delete oid 6 current snap is 16 2024-09-16T04:42:18.949 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:18.947+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:18.988 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:18.988 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:18.988 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:18.988 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:18.988 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:225: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:42:19.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:42:19.050 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:19.047+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:19.132 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:19.132 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:19.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:19.132 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:19.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:147 3.4s0 3:2b4d2b94:::smithi16530038-49:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.197684+0000", 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.931069108000003, 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30111141699999999, 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:19.133 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 147 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.197684+0000", 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.197684+0000", 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.197686+0000", 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.243e-06 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198201+0000", 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00051554599999999997 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198217+0000", 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5335000000000001e-05 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198224+0000", 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5240000000000003e-06 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.294791+0000", 2024-09-16T04:42:19.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.096567302999999993 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.294894+0000", 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000102648 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.295694+0000", 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00079964299999999995 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.296873+0000", 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011794749999999999 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.498721+0000", 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20184827699999999 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.498796+0000", 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4423000000000002e-05 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:148 3.4s0 3:2b4d2b94:::smithi16530038-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.930514684999999, 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.49811553600000003, 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.139 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 148 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198240+0000", 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3600000000000003e-07 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198243+0000", 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.281e-06 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198250+0000", 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7870000000000004e-06 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298324+0000", 2024-09-16T04:42:19.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100073891 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298376+0000", 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2790000000000001e-05 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298437+0000", 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0550999999999997e-05 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298723+0000", 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000285741 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.299721+0000", 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00099841899999999996 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696249+0000", 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39652776099999998 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696354+0000", 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105679 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.145 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:149 3.4s0 3:2b4d2b94:::smithi16530038-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.830402206999999, 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.59985384399999997, 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:19.146 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 149 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.147 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298354+0000", 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298362+0000", 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3319999999999992e-06 2024-09-16T04:42:19.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298377+0000", 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5387999999999999e-05 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298388+0000", 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0957e-05 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398555+0000", 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016630999999999 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398706+0000", 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015114699999999999 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.150 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696493+0000", 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29778684 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696643+0000", 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015093699999999999 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.898205+0000", 2024-09-16T04:42:19.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20156174700000001 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:160 3.4s0 3:2d731f27:::smithi16530038-30:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.674772140999998, 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44291274600000002, 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.152 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 160 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:19.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453983+0000", 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4729999999999999e-06 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453989+0000", 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1299999999999998e-06 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454005+0000", 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5880999999999999e-05 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454014+0000", 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1079999999999999e-06 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454090+0000", 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6062999999999993e-05 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454166+0000", 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5772999999999994e-05 2024-09-16T04:42:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454458+0000", 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029229599999999999 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.455106+0000", 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00064800899999999995 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896766+0000", 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44165951199999998 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896894+0000", 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000128501 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 38/32 ECSubWrite(tid=37, reqid=client.4728.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=38'13))", 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.381049+0000", 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.747704937000002, 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.395884454, 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.381049+0000", 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.381049+0000", 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.381050+0000", 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2279999999999999e-06 2024-09-16T04:42:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.381067+0000", 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7149000000000001e-05 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.381090+0000", 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2946e-05 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.381104+0000", 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4204e-05 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.775529+0000", 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39442521000000003 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.775541+0000", 2024-09-16T04:42:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1464999999999999e-05 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.776847+0000", 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001306038 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.776933+0000", 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6213999999999998e-05 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds5 38/32 ECSubWrite(tid=32, reqid=client.4728.0:209, at_version=38'17, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.475261+0000", 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.653493018999999, 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30293589500000001, 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.475261+0000", 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.475261+0000", 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.475262+0000", 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.468e-06 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.475318+0000", 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6264999999999997e-05 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.475333+0000", 2024-09-16T04:42:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4674e-05 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.575448+0000", 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011506000000001 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.675714+0000", 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10026615899999999 2024-09-16T04:42:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.675739+0000", 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4814999999999999e-05 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.778115+0000", 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102376257 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.778196+0000", 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1197000000000002e-05 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.14s0 42/41 ECSubReadReply(tid=26, attrs_read=0))", 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.832621+0000", 2024-09-16T04:42:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.296132201999999, 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30446411000000001, 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832621+0000", 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832621+0000", 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832623+0000", 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.19e-06 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832646+0000", 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3819e-05 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832666+0000", 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9366000000000001e-05 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832678+0000", 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2602999999999999e-05 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.136804+0000", 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.304125747 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.137086+0000", 2024-09-16T04:42:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028138499999999999 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.14s0 42/41 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.296053425, 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60655789999999998, 2024-09-16T04:42:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832701+0000", 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6400000000000001e-07 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832733+0000", 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2382000000000003e-05 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832757+0000", 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3691999999999999e-05 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.932887+0000", 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100130238 2024-09-16T04:42:19.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.438947+0000", 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50605998399999996 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439258+0000", 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031074 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.178 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 42/41 ECSubWriteReply(tid=29, last_complete=42'15, committed=1, applied=1))", 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.037639+0000", 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.091115031000001, 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30118602, 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037639+0000", 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037639+0000", 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037639+0000", 2024-09-16T04:42:19.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7300000000000005e-07 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037645+0000", 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.677e-06 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037679+0000", 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4048999999999999e-05 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037692+0000", 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3043e-05 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.338804+0000", 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30111130899999999 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.338825+0000", 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1069000000000001e-05 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 42/41 ECSubWriteReply(tid=29, last_complete=42'15, committed=1, applied=1))", 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.037847+0000", 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.090906861000001, 2024-09-16T04:42:19.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30081938400000002, 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037847+0000", 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037847+0000", 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037858+0000", 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0789e-05 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037868+0000", 2024-09-16T04:42:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0234000000000001e-05 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037900+0000", 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2582000000000001e-05 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.037922+0000", 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1849e-05 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.338648+0000", 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.300725505 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.338666+0000", 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8425000000000001e-05 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 42/41 ECSubWriteReply(tid=28, last_complete=42'14, committed=1, applied=1))", 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.039246+0000", 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.089507863000001, 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29934159199999999, 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039246+0000", 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039246+0000", 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039247+0000", 2024-09-16T04:42:19.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6360000000000001e-06 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039256+0000", 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4370000000000007e-06 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039281+0000", 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4992000000000001e-05 2024-09-16T04:42:19.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039300+0000", 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9202999999999999e-05 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.338559+0000", 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29925871900000001 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.338587+0000", 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8605000000000001e-05 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 42/41 ECSubWriteReply(tid=29, last_complete=42'15, committed=1, applied=1))", 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.039898+0000", 2024-09-16T04:42:19.192 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.088856077999999, 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29951841400000001, 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039898+0000", 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039898+0000", 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039899+0000", 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.587e-06 2024-09-16T04:42:19.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039907+0000", 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8089999999999993e-06 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.039930+0000", 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.351e-05 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.140039+0000", 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10010905 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.338921+0000", 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.198881417 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.196 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339416+0000", 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00049504099999999999 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 43/41 ECSubWriteReply(tid=31, last_complete=44'17, committed=1, applied=1))", 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.117620+0000", 2024-09-16T04:42:19.197 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.011134005999999, 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.395738072, 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117620+0000", 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117620+0000", 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117621+0000", 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.418e-06 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117633+0000", 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2187999999999999e-05 2024-09-16T04:42:19.199 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117652+0000", 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9184000000000001e-05 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.117663+0000", 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.512433+0000", 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39476975600000003 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.513358+0000", 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00092447600000000003 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.201 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=56, reqid=osd.11.0:56, at_version=44'19, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.009858977, 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39966409400000003, 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6400000000000003e-07 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118922+0000", 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6635000000000002e-05 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118941+0000", 2024-09-16T04:42:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9707999999999998e-05 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118952+0000", 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.039e-05 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.513442+0000", 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39449013100000002 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.513459+0000", 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6748e-05 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.518526+0000", 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0050673970000000004 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.518559+0000", 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2620999999999998e-05 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=59, reqid=osd.11.0:59, at_version=44'21, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.006993533999999, 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.397602065, 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.207 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121761+0000", 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0699999999999996e-07 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121787+0000", 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6227000000000001e-05 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121804+0000", 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6898999999999999e-05 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121815+0000", 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1294000000000001e-05 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.514082+0000", 2024-09-16T04:42:19.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39226631200000001 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.514105+0000", 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3366e-05 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.519327+0000", 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005222122 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.519362+0000", 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5037999999999999e-05 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.14.22:1 3.4s0 3:208a5e75:::smithi16530038-37:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:19.212 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.467828837999999, 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40385833399999999, 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660929+0000", 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7859999999999998e-06 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660942+0000", 2024-09-16T04:42:19.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3526999999999999e-05 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660996+0000", 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3860999999999998e-05 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.761100+0000", 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100103686 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.857146+0000", 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.096046099999999995 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.857321+0000", 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000175465 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.064783+0000", 2024-09-16T04:42:19.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.207461909 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:367 3.14s0 3:28a14731:::smithi16530038-3:head [append 0~376832] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.181449+0000", 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.947304642999999, 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.31017558299999998, 2024-09-16T04:42:19.217 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 367 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.218 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181449+0000", 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181449+0000", 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181452+0000", 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1130000000000001e-06 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.184888+0000", 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003436012 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.184933+0000", 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4571999999999999e-05 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185006+0000", 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3675000000000002e-05 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185720+0000", 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000713676 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185925+0000", 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020547100000000001 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.187313+0000", 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013878059999999999 2024-09-16T04:42:19.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.389122+0000", 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201808657 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.491491+0000", 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102369028 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.491625+0000", 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000133573 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:368 3.14s0 3:28a14731:::smithi16530038-3:head [setxattr _header (55) in=7b] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:19.224 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.943712732000002, 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50583271100000005, 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 368 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.225 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185043+0000", 2024-09-16T04:42:19.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.463e-06 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185054+0000", 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0356e-05 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185080+0000", 2024-09-16T04:42:19.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6519000000000001e-05 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185097+0000", 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.717e-05 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.188751+0000", 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036535249999999999 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.189019+0000", 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026812599999999999 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.229 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.190416+0000", 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013973830000000001 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.389174+0000", 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19875738700000001 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690722+0000", 2024-09-16T04:42:19.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30154792000000002 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690874+0000", 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000151862 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.231 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:369 3.14s0 3:28a14731:::smithi16530038-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.943631015000001, 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.81195188399999996, 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 369 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.232 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185124+0000", 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.33e-06 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185130+0000", 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9070000000000004e-06 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185154+0000", 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4059000000000001e-05 2024-09-16T04:42:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185173+0000", 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8936e-05 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388708+0000", 2024-09-16T04:42:19.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.203534774 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388994+0000", 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028647200000000002 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691636+0000", 2024-09-16T04:42:19.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.302641681 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691904+0000", 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026824200000000002 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.997074+0000", 2024-09-16T04:42:19.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30517048299999999 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 49/32 ECSubWrite(tid=81, reqid=client.4728.0:376, at_version=49'25, trim_to=0'0, roll_forward_to=45'23))", 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.357186+0000", 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.771567221000002, 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.33446446499999999, 2024-09-16T04:42:19.238 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.357186+0000", 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.357186+0000", 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.357187+0000", 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.063e-06 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.357221+0000", 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3858e-05 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.357239+0000", 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7234000000000001e-05 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.241 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.357252+0000", 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3499000000000001e-05 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.490564+0000", 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.1333124 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.242 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.490579+0000", 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4793e-05 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691534+0000", 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200954667 2024-09-16T04:42:19.243 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691651+0000", 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000116951 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.244 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.245 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:19.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:42:19.247 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:19.243+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:19.303 INFO:tasks.rados.rados.0.smithi165.stdout:225: done (3 left) 2024-09-16T04:42:19.303 INFO:tasks.rados.rados.0.smithi165.stdout:231: done (2 left) 2024-09-16T04:42:19.303 INFO:tasks.rados.rados.0.smithi165.stdout:232: copy_from oid 41 from oid 30 current snap is 16 2024-09-16T04:42:19.303 INFO:tasks.rados.rados.0.smithi165.stdout:233: rollback oid 16 current snap is 16 2024-09-16T04:42:19.304 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 9 2024-09-16T04:42:19.304 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 17 current snap is 16 2024-09-16T04:42:19.304 INFO:tasks.rados.rados.0.smithi165.stdout:234: seq_num 98 ranges {0=491520,491520=589824,1081344=557056,1638400=16384} 2024-09-16T04:42:19.311 INFO:tasks.rados.rados.0.smithi165.stdout:234: writing smithi16530038-17 from 0 to 491520 tid 1 2024-09-16T04:42:19.323 INFO:tasks.rados.rados.0.smithi165.stdout:234: writing smithi16530038-17 from 491520 to 1081344 tid 2 2024-09-16T04:42:19.333 INFO:tasks.rados.rados.0.smithi165.stdout:234: writing smithi16530038-17 from 1081344 to 1638400 tid 3 2024-09-16T04:42:19.334 INFO:tasks.rados.rados.0.smithi165.stdout:234: writing smithi16530038-17 from 1638400 to 1654784 tid 4 2024-09-16T04:42:19.334 INFO:tasks.rados.rados.0.smithi165.stdout:235: rollback oid 45 current snap is 16 2024-09-16T04:42:19.334 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 15 2024-09-16T04:42:19.334 INFO:tasks.rados.rados.0.smithi165.stdout:236: snap_remove snap 8 2024-09-16T04:42:19.346 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:19.343+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:19.435 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:19.435 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:19.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.435 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=75, reqid=client.4728.0:577, at_version=69'50, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:19.435 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.105281708, 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.105337477, 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329019+0000", 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.654e-06 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329563+0000", 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054386599999999995 2024-09-16T04:42:19.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329591+0000", 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7759999999999998e-05 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329619+0000", 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7968999999999999e-05 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427652+0000", 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098032790999999994 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427662+0000", 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0736e-05 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=76, reqid=client.4728.0:578, at_version=69'51, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:19.439 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.096174944999999998, 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.096434172999999998, 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338125+0000", 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9099999999999991e-07 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338313+0000", 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018766200000000001 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.441 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338329+0000", 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5807e-05 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338339+0000", 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0451e-05 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=77, reqid=client.4728.0:579, at_version=69'52, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:19.442 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.092342374000000005, 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.092741255999999994, 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.443 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341958+0000", 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0400000000000005e-07 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342250+0000", 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029209100000000002 2024-09-16T04:42:19.444 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342265+0000", 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4808e-05 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342273+0000", 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3760000000000008e-06 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 69/67 ECSubWrite(tid=147, reqid=client.4728.0:574, at_version=69'46, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.406934+0000", 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.027365252999999999, 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027884023000000001, 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.406934+0000", 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.446 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.406934+0000", 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.406935+0000", 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.13e-06 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.406984+0000", 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8405999999999997e-05 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.407017+0000", 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2866e-05 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.407036+0000", 2024-09-16T04:42:19.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9313e-05 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.432504+0000", 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.025467680999999999 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.432520+0000", 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.698e-05 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.449 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.434784+0000", 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0022630810000000001 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 69/67 ECSubWrite(tid=148, reqid=client.4728.0:575, at_version=69'47, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:19.450 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.024043697999999999, 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024669608999999999, 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:19.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410258+0000", 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1299999999999999e-06 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410345+0000", 2024-09-16T04:42:19.452 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6930999999999998e-05 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410382+0000", 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7292999999999997e-05 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410407+0000", 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5029999999999999e-05 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.453 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.454 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.454 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:19.454 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 5 2024-09-16T04:42:19.454 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:19.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:42:19.456 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:19.451+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:19.556 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:19.551+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:19.627 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:19.627 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:19.627 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:19.627 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:19.627 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:19.641 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:42:19.685 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:19.683+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:19.758 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:19.755+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 37/32 ECSubWrite(tid=34, reqid=client.4728.0:169, at_version=37'11, trim_to=0'0, roll_forward_to=36'10))", 2024-09-16T04:42:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:47.573967+0000", 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.185449487, 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102186843, 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.573967+0000", 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.573967+0000", 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.573969+0000", 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0540000000000002e-06 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.574032+0000", 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2860999999999993e-05 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.574074+0000", 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1928000000000002e-05 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.574098+0000", 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4189000000000002e-05 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.674354+0000", 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100256153 2024-09-16T04:42:19.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.674369+0000", 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5342e-05 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.676085+0000", 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001715762 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.676154+0000", 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8553999999999994e-05 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 38/32 ECSubWrite(tid=33, reqid=client.4728.0:181, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.272884+0000", 2024-09-16T04:42:19.767 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 31.486532043, 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102436214, 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272884+0000", 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272884+0000", 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272886+0000", 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.273116+0000", 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000229931 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.273147+0000", 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1699000000000003e-05 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.273172+0000", 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4389e-05 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.373657+0000", 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10048555200000001 2024-09-16T04:42:19.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.373667+0000", 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8009999999999998e-06 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.375271+0000", 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001603527 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.375320+0000", 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9795e-05 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 38/32 ECSubWrite(tid=34, reqid=client.4728.0:182, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.277177+0000", 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 31.482239358000001, 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101968824, 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.772 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277177+0000", 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277177+0000", 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277179+0000", 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9970000000000001e-06 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277391+0000", 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021181499999999999 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.277441+0000", 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0430999999999999e-05 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377525+0000", 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10008368300000001 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377604+0000", 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9330000000000001e-05 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.377613+0000", 2024-09-16T04:42:19.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6659999999999997e-06 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.379126+0000", 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015134510000000001 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.379146+0000", 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9451000000000001e-05 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=64, reqid=osd.11.0:59, at_version=0'0, trim_to=0'0, roll_forward_to=44'21))", 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.520506+0000", 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.238910121, 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10161100200000001, 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520506+0000", 2024-09-16T04:42:19.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520506+0000", 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520507+0000", 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8100000000000001e-07 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520520+0000", 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2449999999999999e-05 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.520535+0000", 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5264000000000002e-05 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.620622+0000", 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100087539 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.620782+0000", 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015924600000000001 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.620812+0000", 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0015999999999999e-05 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.622052+0000", 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012404619999999999 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.622117+0000", 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5043999999999996e-05 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=29, reqid=client.4728.0:365, at_version=49'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.488007855999999, 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15627164299999999, 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271409+0000", 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271765+0000", 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00035584100000000002 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271795+0000", 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9595999999999999e-05 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271820+0000", 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5301999999999999e-05 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425575+0000", 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15375470799999999 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425593+0000", 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427644+0000", 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020504630000000002 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427680+0000", 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6613999999999999e-05 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 50/32 ECSubWrite(tid=35, reqid=client.4728.0:386, at_version=50'18, trim_to=0'0, roll_forward_to=49'15))", 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:00.209177+0000", 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.550238882999999, 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101895481, 2024-09-16T04:42:19.786 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209177+0000", 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209177+0000", 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209178+0000", 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3900000000000003e-07 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209453+0000", 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027417499999999998 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.209471+0000", 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8575000000000001e-05 2024-09-16T04:42:19.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.309558+0000", 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100086831 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.309617+0000", 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8621e-05 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.309623+0000", 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7580000000000003e-06 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.311032+0000", 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001408833 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:00.311073+0000", 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0748999999999999e-05 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.790 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/32 ECSubWrite(tid=86, reqid=client.4728.0:424, at_version=53'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 16.419900174999999, 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10390139399999999, 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339517+0000", 2024-09-16T04:42:19.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.167e-06 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339944+0000", 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00042670199999999999 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339971+0000", 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6916000000000001e-05 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339982+0000", 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1445e-05 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.440244+0000", 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100261356 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.440271+0000", 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7209e-05 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.443349+0000", 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030780949999999999 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.443418+0000", 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8503999999999993e-05 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/32 ECSubWrite(tid=90, reqid=client.4728.0:427, at_version=53'30, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.936308+0000", 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.823108327, 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10156912, 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.936308+0000", 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.936308+0000", 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.936310+0000", 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.607e-06 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.936342+0000", 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2571999999999999e-05 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.936373+0000", 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0852999999999998e-05 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.036510+0000", 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10013699 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.036625+0000", 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000115133 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.036644+0000", 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8754e-05 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.037801+0000", 2024-09-16T04:42:19.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011574560000000001 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.037877+0000", 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5754999999999996e-05 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=118, reqid=client.4728.0:458, at_version=58'27, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:19.800 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.354001844000001, 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20346340800000001, 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.801 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405416+0000", 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.221e-06 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405551+0000", 2024-09-16T04:42:19.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013492800000000001 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405567+0000", 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6552999999999999e-05 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505670+0000", 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10010258399999999 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505840+0000", 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017047199999999999 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505892+0000", 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1406000000000002e-05 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608838+0000", 2024-09-16T04:42:19.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10294645299999999 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608878+0000", 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9790999999999999e-05 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=119, reqid=client.4728.0:459, at_version=58'28, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.505750+0000", 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.253666027, 2024-09-16T04:42:19.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10316181200000001, 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505750+0000", 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505750+0000", 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505752+0000", 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5069999999999999e-06 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505837+0000", 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5020999999999998e-05 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505877+0000", 2024-09-16T04:42:19.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0028999999999999e-05 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505901+0000", 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3833e-05 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507589+0000", 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00168877 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507620+0000", 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0366e-05 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608884+0000", 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101264361 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608912+0000", 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7925000000000001e-05 2024-09-16T04:42:19.809 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=120, reqid=client.4728.0:460, at_version=58'29, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.253471091, 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10344765, 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.810 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505946+0000", 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:19.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505979+0000", 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3062000000000003e-05 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.506027+0000", 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7333000000000002e-05 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.506050+0000", 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2918999999999999e-05 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608654+0000", 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102604337 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608663+0000", 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.986e-06 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.609362+0000", 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00069878900000000001 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.609393+0000", 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1241000000000001e-05 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 59/32 ECSubWrite(tid=119, reqid=osd.11.0:119, at_version=59'48, trim_to=0'0, roll_forward_to=58'44))", 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:09.238987+0000", 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.520429479000001, 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102666459, 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.238987+0000", 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.238987+0000", 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.238993+0000", 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0090000000000004e-06 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.239025+0000", 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2252999999999998e-05 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.239046+0000", 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0738999999999999e-05 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.339146+0000", 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100099935 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.339239+0000", 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3381000000000006e-05 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.339249+0000", 2024-09-16T04:42:19.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.028e-05 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.341604+0000", 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023544400000000002 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.341653+0000", 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9422000000000002e-05 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.822 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/32 ECSubWrite(tid=132, reqid=osd.4.0:132, at_version=60'35, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.2965477029999999, 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20257109500000001, 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.823 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3300000000000003e-07 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.824 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462925+0000", 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6362e-05 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462958+0000", 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2876000000000002e-05 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462991+0000", 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2836000000000002e-05 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.825 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463182+0000", 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000190736 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463210+0000", 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7925000000000001e-05 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665388+0000", 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202177734 2024-09-16T04:42:19.826 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665440+0000", 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2193e-05 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.2955894640000007, 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20164931799999999, 2024-09-16T04:42:19.827 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.828 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463828+0000", 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.161e-06 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463862+0000", 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3846000000000001e-05 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463881+0000", 2024-09-16T04:42:19.829 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9596e-05 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463895+0000", 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3448999999999999e-05 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.464025+0000", 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000129677 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.464038+0000", 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3654e-05 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665451+0000", 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201412806 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.831 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665476+0000", 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5128999999999999e-05 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 64/32 ECSubWrite(tid=57, reqid=client.4728.0:507, at_version=0'0, trim_to=0'0, roll_forward_to=64'36))", 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:14.799260+0000", 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9601561519999997, 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101527573, 2024-09-16T04:42:19.832 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.799260+0000", 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.799260+0000", 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.833 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.799262+0000", 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.75e-06 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.799283+0000", 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1172999999999999e-05 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.799313+0000", 2024-09-16T04:42:19.834 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0088999999999999e-05 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.799333+0000", 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9930000000000001e-05 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.899585+0000", 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100251889 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.899608+0000", 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3476e-05 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.900688+0000", 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010799850000000001 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:14.900788+0000", 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9281e-05 2024-09-16T04:42:19.836 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 65/32 ECSubWrite(tid=59, reqid=osd.2.0:59, at_version=65'40, trim_to=0'0, roll_forward_to=64'36))", 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.3074364110000003, 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202052495, 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.838 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451982+0000", 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.227e-06 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452027+0000", 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5142000000000002e-05 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452061+0000", 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3529999999999999e-05 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452079+0000", 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7941999999999999e-05 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.652452+0000", 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200372941 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.840 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.652468+0000", 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6517000000000001e-05 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.653953+0000", 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001484857 2024-09-16T04:42:19.841 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.654032+0000", 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9338999999999994e-05 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=75, reqid=client.4728.0:577, at_version=69'50, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:19.842 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.43039867199999998, 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20012123100000001, 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:19.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.844 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.844 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.844 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.844 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329019+0000", 2024-09-16T04:42:19.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.654e-06 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329563+0000", 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054386599999999995 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329591+0000", 2024-09-16T04:42:19.845 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7759999999999998e-05 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329619+0000", 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7968999999999999e-05 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.846 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427652+0000", 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098032790999999994 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427662+0000", 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0736e-05 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529104+0000", 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10144188 2024-09-16T04:42:19.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529139+0000", 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4575e-05 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.848 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=76, reqid=client.4728.0:578, at_version=69'51, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.42129190900000002, 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19168868999999999, 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338125+0000", 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9099999999999991e-07 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338313+0000", 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018766200000000001 2024-09-16T04:42:19.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338329+0000", 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5807e-05 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338339+0000", 2024-09-16T04:42:19.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0451e-05 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.528499+0000", 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19015979599999999 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.528508+0000", 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0469999999999999e-06 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529785+0000", 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001276758 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529813+0000", 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8178e-05 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=77, reqid=client.4728.0:579, at_version=69'52, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.41745933800000001, 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18830155900000001, 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.855 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.856 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341958+0000", 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0400000000000005e-07 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342250+0000", 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029209100000000002 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342265+0000", 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4808e-05 2024-09-16T04:42:19.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342273+0000", 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3760000000000008e-06 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529207+0000", 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18693383999999999 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529216+0000", 2024-09-16T04:42:19.858 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9109999999999999e-06 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.530224+0000", 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00100815 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.530259+0000", 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4579000000000002e-05 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.859 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 69/67 ECSubWrite(tid=148, reqid=client.4728.0:575, at_version=69'47, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.34916066200000001, 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102672033, 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:19.860 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410258+0000", 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1299999999999999e-06 2024-09-16T04:42:19.861 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410345+0000", 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6930999999999998e-05 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410382+0000", 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7292999999999997e-05 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410407+0000", 2024-09-16T04:42:19.862 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5029999999999999e-05 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511669+0000", 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101262251 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511687+0000", 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7390000000000001e-05 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.863 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.512888+0000", 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012010910000000001 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.512928+0000", 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9916999999999997e-05 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:19.864 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:19.866 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:42:19.867 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:19.863+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:19.967 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:19.963+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:19.998 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:19.998 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:19.998 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:19.998 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:20.012 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:42:20.069 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:20.063+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:20.133 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:20.133 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:20.137 INFO:tasks.rados.rados.0.smithi165.stdout:236: done (6 left) 2024-09-16T04:42:20.137 INFO:tasks.rados.rados.0.smithi165.stdout:237: snap_remove snap 13 2024-09-16T04:42:20.157 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:42:20.169 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:20.164+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:20.205 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:42:20.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T04:42:20.270 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:20.267+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:20.278 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:20.278 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:20.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:20.279 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:20.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.279 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=10, reqid=client.4728.0:122, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:20.279 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:20.279 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 34.592252389999999, 2024-09-16T04:42:20.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201524707, 2024-09-16T04:42:20.280 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.280 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.280 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.280 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.280 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.280 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684051+0000", 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.277e-06 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684081+0000", 2024-09-16T04:42:20.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0414999999999999e-05 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684109+0000", 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7455000000000001e-05 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684125+0000", 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5707000000000001e-05 2024-09-16T04:42:20.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884504+0000", 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200379487 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884519+0000", 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4953000000000001e-05 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885504+0000", 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098537600000000005 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885574+0000", 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0036999999999994e-05 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 34.301930790999997, 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20260964000000001, 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.286 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974373+0000", 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.683e-06 2024-09-16T04:42:20.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974841+0000", 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468255 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974891+0000", 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0198000000000003e-05 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.074965+0000", 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100073226 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175170+0000", 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100205119 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175184+0000", 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4626e-05 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176898+0000", 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017136390000000001 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176981+0000", 2024-09-16T04:42:20.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2893999999999996e-05 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=16, reqid=client.4728.0:131, at_version=36'4, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 34.201287501000003, 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102541787, 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.291 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075015+0000", 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075016+0000", 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.06e-06 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075046+0000", 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0485999999999999e-05 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075066+0000", 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9572000000000002e-05 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.075078+0000", 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2626000000000001e-05 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176189+0000", 2024-09-16T04:42:20.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10111062699999999 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176204+0000", 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4543e-05 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.177494+0000", 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012900450000000001 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.177556+0000", 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2828000000000002e-05 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:160, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 33.822639181, 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20191688999999999, 2024-09-16T04:42:20.296 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.297 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453664+0000", 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453695+0000", 2024-09-16T04:42:20.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0998999999999999e-05 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453714+0000", 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9745000000000002e-05 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.553888+0000", 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10017351100000001 2024-09-16T04:42:20.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654137+0000", 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100248949 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654151+0000", 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4279e-05 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655526+0000", 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013750520000000001 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655580+0000", 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3634999999999997e-05 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s1 36/32 ECSubWrite(tid=22, reqid=client.4728.0:157, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 33.690936839000003, 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102232006, 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.302 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585365+0000", 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585367+0000", 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585532+0000", 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164928 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585561+0000", 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9391e-05 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.585578+0000", 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6569999999999999e-05 2024-09-16T04:42:20.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.685858+0000", 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100280496 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.685876+0000", 2024-09-16T04:42:20.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7499999999999998e-05 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.687532+0000", 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016559509999999999 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.687597+0000", 2024-09-16T04:42:20.307 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5456999999999993e-05 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s3 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.409215+0000", 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 27.867086721, 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10267963099999999, 2024-09-16T04:42:20.308 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409215+0000", 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409215+0000", 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409217+0000", 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4869999999999999e-06 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409262+0000", 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5488999999999998e-05 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409289+0000", 2024-09-16T04:42:20.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6682e-05 2024-09-16T04:42:20.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409305+0000", 2024-09-16T04:42:20.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5976000000000001e-05 2024-09-16T04:42:20.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.509553+0000", 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100247613 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.509580+0000", 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6872e-05 2024-09-16T04:42:20.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.511814+0000", 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022344019999999999 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.511895+0000", 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1110000000000001e-05 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.19s1 42/41 ECSubRead(tid=32, to_read={3:9e9173cd:::smithi16530038-4:head=0,16384,0}, subchunks={3:9e9173cd:::smithi16530038-4:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.414144+0000", 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 27.862157836000002, 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.1022832, 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414144+0000", 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414144+0000", 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414145+0000", 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.023e-06 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414159+0000", 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3359e-05 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414186+0000", 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7118999999999999e-05 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516004+0000", 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101818013 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516187+0000", 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018360100000000001 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516427+0000", 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024008499999999999 2024-09-16T04:42:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.19s1 42/41 ECSubWrite(tid=35, reqid=client.4728.0:262, at_version=42'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.517939+0000", 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 27.758363264, 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103147299, 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517939+0000", 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517939+0000", 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517940+0000", 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9699999999999995e-07 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.517999+0000", 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9308000000000002e-05 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518020+0000", 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1305999999999999e-05 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518040+0000", 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0123e-05 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518525+0000", 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00048423699999999999 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518546+0000", 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1265999999999999e-05 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.620987+0000", 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102440958 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621086+0000", 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9303999999999997e-05 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.19s1 42/41 ECSubWrite(tid=36, reqid=client.4728.0:263, at_version=42'14, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.518116+0000", 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 27.758186002999999, 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10640616899999999, 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518116+0000", 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518116+0000", 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518117+0000", 2024-09-16T04:42:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7300000000000001e-07 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518186+0000", 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8981999999999998e-05 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518208+0000", 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2705e-05 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.518228+0000", 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0129999999999999e-05 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619775+0000", 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101546368 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.329 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619800+0000", 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4879999999999999e-05 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.624447+0000", 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046478400000000003 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.624522+0000", 2024-09-16T04:42:20.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4790999999999996e-05 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s3 42/41 ECSubWrite(tid=23, reqid=client.4728.0:197, at_version=42'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 27.757247534000001, 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102207279, 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.331 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519055+0000", 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519168+0000", 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113026 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519192+0000", 2024-09-16T04:42:20.333 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3189000000000001e-05 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.519208+0000", 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6725999999999999e-05 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619474+0000", 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100266022 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.334 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619502+0000", 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7429000000000001e-05 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621190+0000", 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016884529999999999 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621262+0000", 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1643999999999994e-05 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.es2 43/32 ECSubRead(tid=21, to_read={3:741a05b8:::smithi16530038-36:head=0,147456,0}, subchunks={3:741a05b8:::smithi16530038-36:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:53.203899+0000", 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 27.072403129000001, 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102872009, 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.336 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203899+0000", 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203899+0000", 2024-09-16T04:42:20.337 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203901+0000", 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.807e-06 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203917+0000", 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5953000000000001e-05 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.203949+0000", 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2149e-05 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.304066+0000", 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10011745399999999 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.306455+0000", 2024-09-16T04:42:20.340 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002388879 2024-09-16T04:42:20.340 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.340 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.340 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.340 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.306771+0000", 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000315767 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.as2 43/32 ECSubRead(tid=37, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 27.067887458000001, 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200544685, 2024-09-16T04:42:20.341 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.34e-07 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208425+0000", 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8859999999999999e-06 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208457+0000", 2024-09-16T04:42:20.344 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1644999999999999e-05 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.308556+0000", 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100099178 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.408782+0000", 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100226105 2024-09-16T04:42:20.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.408959+0000", 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017743699999999999 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 48/32 ECSubWrite(tid=44, reqid=client.4728.0:362, at_version=48'17, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 21.919615880999999, 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205456467, 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.347 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356687+0000", 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.349 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356985+0000", 2024-09-16T04:42:20.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029765100000000001 2024-09-16T04:42:20.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.350 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357014+0000", 2024-09-16T04:42:20.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9853e-05 2024-09-16T04:42:20.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.351 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357034+0000", 2024-09-16T04:42:20.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9592000000000001e-05 2024-09-16T04:42:20.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.352 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557308+0000", 2024-09-16T04:42:20.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20027399500000001 2024-09-16T04:42:20.352 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.352 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.352 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557322+0000", 2024-09-16T04:42:20.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4346e-05 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562072+0000", 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047490960000000004 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562143+0000", 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1191000000000003e-05 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.353 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.2s5 53/37 ECSubWrite(tid=192, reqid=client.4728.0:420, at_version=53'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:03.322544+0000", 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.953757717999999, 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102605686, 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.354 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322544+0000", 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322544+0000", 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322546+0000", 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322915+0000", 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00036882700000000001 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.356 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.322956+0000", 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0926000000000001e-05 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.423060+0000", 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100104049 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.423202+0000", 2024-09-16T04:42:20.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141903 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.423219+0000", 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6849000000000001e-05 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.425059+0000", 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0018400280000000001 2024-09-16T04:42:20.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:03.425150+0000", 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1124000000000004e-05 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s6 57/53 ECSubWrite(tid=9, reqid=osd.10.0:9, at_version=57'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:20.359 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.047184974, 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209615832, 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.360 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229119+0000", 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229208+0000", 2024-09-16T04:42:20.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9190999999999994e-05 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229249+0000", 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0842e-05 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329348+0000", 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10009905600000001 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426747+0000", 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.09739892 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426767+0000", 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0173000000000002e-05 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438659+0000", 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011892171 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438733+0000", 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3515000000000003e-05 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.364 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s6 57/53 ECSubWrite(tid=10, reqid=osd.10.0:10, at_version=57'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.329396+0000", 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.946906054999999, 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.109377182, 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329396+0000", 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329396+0000", 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329397+0000", 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2470000000000001e-06 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329438+0000", 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1118999999999997e-05 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329462+0000", 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3178999999999999e-05 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329475+0000", 2024-09-16T04:42:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3873000000000001e-05 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.427340+0000", 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097864638000000004 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.427353+0000", 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3380000000000001e-05 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438740+0000", 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011386240000000001 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438773+0000", 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3506000000000001e-05 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.bs2 58/32 ECSubWrite(tid=115, reqid=client.4728.0:457, at_version=58'26, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.329739+0000", 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.946563086999999, 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10415880299999999, 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329739+0000", 2024-09-16T04:42:20.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329739+0000", 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329739+0000", 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7399999999999998e-07 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329935+0000", 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000195892 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329955+0000", 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9148999999999999e-05 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329968+0000", 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3923e-05 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.430193+0000", 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10022435 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.430223+0000", 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0088999999999999e-05 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.433830+0000", 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003606761 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.433898+0000", 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8164999999999995e-05 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.2s5 59/37 ECSubWrite(tid=210, reqid=osd.13.0:207, at_version=0'0, trim_to=0'0, roll_forward_to=59'15))", 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:09.761991+0000", 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.514310719999999, 2024-09-16T04:42:20.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102243187, 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.761991+0000", 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.761991+0000", 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.761993+0000", 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.933e-06 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.762024+0000", 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0222999999999998e-05 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.762071+0000", 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7584999999999999e-05 2024-09-16T04:42:20.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.762095+0000", 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3969999999999999e-05 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.862303+0000", 2024-09-16T04:42:20.379 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10020777 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.862319+0000", 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6062999999999999e-05 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.380 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.864133+0000", 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001813582 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.864235+0000", 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000102061 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s3 63/55 ECSubWrite(tid=22, reqid=osd.5.0:22, at_version=63'14, trim_to=0'0, roll_forward_to=62'12))", 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.6979058330000001, 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20251555199999999, 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.382 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.383 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:20.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:20.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9999999999999999e-07 2024-09-16T04:42:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578420+0000", 2024-09-16T04:42:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3271000000000001e-05 2024-09-16T04:42:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578449+0000", 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9488999999999998e-05 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.678595+0000", 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100145814 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.386 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778824+0000", 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100228703 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778840+0000", 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6123000000000001e-05 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780838+0000", 2024-09-16T04:42:20.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019980129999999999 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780912+0000", 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3938999999999998e-05 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es2 69/67 ECSubWrite(tid=80, reqid=client.4728.0:581, at_version=0'0, trim_to=0'0, roll_forward_to=69'54))", 2024-09-16T04:42:20.388 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.54973112000000002, 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20154364599999999, 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726573+0000", 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1399999999999998e-06 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726628+0000", 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4376000000000003e-05 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726668+0000", 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0466000000000002e-05 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.391 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.826832+0000", 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100163697 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827016+0000", 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018461700000000001 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827048+0000", 2024-09-16T04:42:20.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2186e-05 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928006+0000", 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10095765900000001 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928115+0000", 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010850500000000001 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.393 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.394 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.394 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:20.394 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:42:20.396 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:42:20.397 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:20.391+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:20.496 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:20.491+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:20.581 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s3 70/67 ECSubWrite(tid=154, reqid=osd.4.0:151, at_version=0'0, trim_to=0'0, roll_forward_to=70'33))", 2024-09-16T04:42:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:20.566073+0000", 2024-09-16T04:42:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.01407718, 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014115473, 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "no flag points reached", 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.566073+0000", 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.566073+0000", 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.566074+0000", 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0200000000000002e-07 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.566085+0000", 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0762e-05 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.566097+0000", 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 70/55 ECSubWriteReply(tid=34, last_complete=70'21, committed=1, applied=1))", 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:20.567274+0000", 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.012876830000000001, 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012993078, 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.567274+0000", 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.567274+0000", 2024-09-16T04:42:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.567275+0000", 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1699999999999997e-07 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.567279+0000", 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8849999999999998e-06 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.567295+0000", 2024-09-16T04:42:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5809999999999999e-05 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.567304+0000", 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5380000000000002e-06 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 2 2024-09-16T04:42:20.589 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:20.595 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:42:20.597 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:20.591+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:20.698 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:20.695+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:20.720 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:20.720 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:20.720 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:20.720 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:20.720 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:20.734 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:42:20.799 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:20.796+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:20.809 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:20.862 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4356.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.460987084999999, 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31207237399999999, 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.863 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4356", 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.59:0/2607617674", 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.864 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398126+0000", 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0999999999999998e-06 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398137+0000", 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0253e-05 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398158+0000", 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1401999999999999e-05 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398176+0000", 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8552000000000001e-05 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597014+0000", 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19883803 2024-09-16T04:42:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597043+0000", 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8683999999999998e-05 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.604764+0000", 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0077206829999999999 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605002+0000", 2024-09-16T04:42:20.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023771400000000001 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 15", 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605234+0000", 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023226099999999999 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.606579+0000", 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013450650000000001 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710048+0000", 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103468869 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710127+0000", 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9491000000000002e-05 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710197+0000", 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9270000000000006e-05 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.869 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:190 3.8s0 3:16f2ba44:::smithi16530038-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.759002907999999, 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70466111799999998, 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:20.870 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 190 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100109+0000", 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7300000000000004e-07 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100116+0000", 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3099999999999997e-06 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100128+0000", 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2262000000000001e-05 2024-09-16T04:42:20.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100137+0000", 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480895+0000", 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.38075793800000002 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480902+0000", 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0700000000000001e-06 2024-09-16T04:42:20.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582260+0000", 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101357841 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582314+0000", 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4175000000000002e-05 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804615+0000", 2024-09-16T04:42:20.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.22230154099999999 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804681+0000", 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5356000000000002e-05 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804769+0000", 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8782000000000005e-05 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 28.449738121999999, 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.311529638, 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409375+0000", 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8899999999999999e-06 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409439+0000", 2024-09-16T04:42:20.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4428e-05 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409480+0000", 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0454999999999998e-05 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409505+0000", 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4620000000000001e-05 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619957+0000", 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21045218900000001 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619979+0000", 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1866e-05 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720830+0000", 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100851396 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720903+0000", 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2793999999999995e-05 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=31, reqid=client.4728.0:186, at_version=42'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 28.444453246999998, 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.306308988, 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.880 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414660+0000", 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414693+0000", 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414716+0000", 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3356000000000001e-05 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414734+0000", 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7787000000000001e-05 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621156+0000", 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.206422364 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621180+0000", 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3589999999999999e-05 2024-09-16T04:42:20.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720911+0000", 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099731400999999997 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720967+0000", 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5841000000000003e-05 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=33, reqid=client.4728.0:187, at_version=42'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.516721+0000", 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 28.342390478999999, 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.204424734, 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516721+0000", 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516721+0000", 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516723+0000", 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1069999999999999e-06 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516769+0000", 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5945000000000001e-05 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516801+0000", 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2289000000000002e-05 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.516825+0000", 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3405999999999999e-05 2024-09-16T04:42:20.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.719877+0000", 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20305267599999999 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.719903+0000", 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6100000000000001e-05 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.721087+0000", 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00118366 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.721146+0000", 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8551000000000002e-05 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:20.889 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.494732403, 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30496093299999999, 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:20.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364381+0000", 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.663e-06 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364619+0000", 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023814999999999999 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364642+0000", 2024-09-16T04:42:20.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2875999999999999e-05 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364655+0000", 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3849000000000001e-05 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.667942+0000", 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30328671699999998 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.669340+0000", 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013976780000000001 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s5 47/37 ECSubWrite(tid=189, reqid=osd.13.0:189, at_version=47'9, trim_to=0'0, roll_forward_to=46'7))", 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.438051+0000", 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.421060463, 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20431723199999999, 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.893 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438051+0000", 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438051+0000", 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438052+0000", 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3399999999999998e-07 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438101+0000", 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8952000000000002e-05 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438129+0000", 2024-09-16T04:42:20.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8498999999999999e-05 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.438153+0000", 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3558999999999999e-05 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.638528+0000", 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200375478 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.638544+0000", 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.596e-05 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.642277+0000", 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037327660000000002 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.642368+0000", 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1184e-05 2024-09-16T04:42:20.897 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s4 57/53 ECSubWrite(tid=10, reqid=osd.10.0:10, at_version=57'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.231241+0000", 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.627870397000001, 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20220089499999999, 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.231241+0000", 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.231241+0000", 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.231243+0000", 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8029999999999999e-06 2024-09-16T04:42:20.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.231294+0000", 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1309e-05 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.231325+0000", 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1408999999999997e-05 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.231349+0000", 2024-09-16T04:42:20.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3609999999999999e-05 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.431633+0000", 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20028389699999999 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.431664+0000", 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0954999999999998e-05 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.433370+0000", 2024-09-16T04:42:20.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001706227 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.433442+0000", 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1685000000000002e-05 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 60/32 ECSubWrite(tid=123, reqid=osd.11.0:123, at_version=60'50, trim_to=0'0, roll_forward_to=59'48))", 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.3706286, 2024-09-16T04:42:20.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30259428900000002, 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5400000000000002e-07 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488522+0000", 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9271999999999999e-05 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488552+0000", 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9745999999999999e-05 2024-09-16T04:42:20.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588697+0000", 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100144495 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.789105+0000", 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20040802099999999 2024-09-16T04:42:20.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.789120+0000", 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5818999999999999e-05 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.791010+0000", 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001889944 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.791077+0000", 2024-09-16T04:42:20.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6637999999999995e-05 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 60/32 ECSubWrite(tid=124, reqid=osd.11.0:124, at_version=60'52, trim_to=0'0, roll_forward_to=59'48))", 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:10.588735+0000", 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.270376717, 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.204080393, 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.907 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588735+0000", 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588735+0000", 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588735+0000", 2024-09-16T04:42:20.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.33e-07 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588774+0000", 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8439000000000002e-05 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588798+0000", 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4012999999999999e-05 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588814+0000", 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6443000000000002e-05 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.789722+0000", 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20090756900000001 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.789739+0000", 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6809000000000001e-05 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.792705+0000", 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0029664930000000002 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.792815+0000", 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000109694 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:523 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:20.911 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.6259899940000002, 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41544228799999999, 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 523 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9069999999999999e-06 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233135+0000", 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1303e-05 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233167+0000", 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1415999999999999e-05 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233183+0000", 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6495999999999999e-05 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.239234+0000", 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006050153 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.239492+0000", 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000258872 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.648564+0000", 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40907114100000003 2024-09-16T04:42:20.915 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:524 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.625901153, 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41687002699999998, 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:20.916 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 524 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233211+0000", 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0100000000000001e-06 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233233+0000", 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1647000000000001e-05 2024-09-16T04:42:20.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233246+0000", 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.332e-05 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233256+0000", 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7319999999999993e-06 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.240876+0000", 2024-09-16T04:42:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076197690000000002 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241108+0000", 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000232103 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.650080+0000", 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40897244599999999 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:525 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.6258382500000002, 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41781336400000002, 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 525 2024-09-16T04:42:20.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233274+0000", 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1999999999999998e-07 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233279+0000", 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6799999999999998e-06 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233293+0000", 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3169e-05 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233303+0000", 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0540000000000001e-05 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.343879+0000", 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11057620899999999 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344065+0000", 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000186083 2024-09-16T04:42:20.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.651086+0000", 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30702086299999998 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.17s0 65/53 ECSubRead(tid=25, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.6174750099999997, 2024-09-16T04:42:20.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30542288899999998, 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241846+0000", 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021020300000000001 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241873+0000", 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6367e-05 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.546861+0000", 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30498862300000001 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.547059+0000", 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197696 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 64/53 ECSubReadReply(tid=24, attrs_read=0))", 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.241857+0000", 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.6172537699999996, 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.204620895, 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.929 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241857+0000", 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241857+0000", 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241859+0000", 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.846e-06 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.242133+0000", 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027401500000000001 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.242164+0000", 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0185999999999999e-05 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.242178+0000", 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4100000000000001e-05 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.446393+0000", 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20421528 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.446478+0000", 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5468000000000002e-05 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 64/53 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-16T04:42:20.933 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.6101768190000003, 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.39834076099999999, 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248935+0000", 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249102+0000", 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166933 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249127+0000", 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249156+0000", 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8572999999999999e-05 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647206+0000", 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.39805037799999998 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647275+0000", 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8837999999999994e-05 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 65/53 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.5172693050000001, 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30554653199999998, 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.938 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341843+0000", 2024-09-16T04:42:20.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.533e-06 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342145+0000", 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00030105099999999999 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342175+0000", 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0159e-05 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342192+0000", 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7003e-05 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647333+0000", 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30514099700000002 2024-09-16T04:42:20.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647388+0000", 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5788999999999997e-05 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.17s0 65/53 ECSubRead(tid=26, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:20.942 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.5146366469999997, 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30318800499999998, 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344537+0000", 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2811999999999994e-05 2024-09-16T04:42:20.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.946 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.946 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344572+0000", 2024-09-16T04:42:20.946 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4798999999999997e-05 2024-09-16T04:42:20.946 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.946 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647490+0000", 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30291773100000002 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647663+0000", 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017266300000000001 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.947 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 65/53 ECSubReadReply(tid=26, attrs_read=0))", 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.348140+0000", 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.5109711130000001, 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.300759887, 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348140+0000", 2024-09-16T04:42:20.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348140+0000", 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348142+0000", 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3199999999999998e-06 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348447+0000", 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00030485400000000002 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348483+0000", 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5716999999999997e-05 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348503+0000", 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9969000000000001e-05 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.648801+0000", 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30029768200000001 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.648900+0000", 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9345000000000005e-05 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 65/53 ECSubReadReply(tid=24, attrs_read=0))", 2024-09-16T04:42:20.951 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.348554+0000", 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.5105569479999996, 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30004328400000002, 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348554+0000", 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348554+0000", 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348556+0000", 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.217e-06 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348806+0000", 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00025038799999999999 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.348837+0000", 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1433999999999998e-05 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.448973+0000", 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100135455 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647711+0000", 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.198738468 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.648598+0000", 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00088632200000000004 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:20.955 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:20.957 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:42:20.961 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:20.951+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:20.967 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:20.989 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:20.989 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:21.010 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:21.010 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:21.033 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:21.033 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:21.054 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:21.057 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:21.051+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:21.073 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:21.073 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:21.088 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:21.088 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:21.089 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:21.089 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:21.095 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:21.095 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:21.103 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:42:21.118 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:21.135 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T04:42:21.135 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:42:21.139 INFO:tasks.rados.rados.0.smithi165.stdout:237: done (6 left) 2024-09-16T04:42:21.139 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 49 current snap is 16 2024-09-16T04:42:21.140 INFO:tasks.rados.rados.0.smithi165.stdout:232: finishing copy_from to smithi16530038-41 2024-09-16T04:42:21.140 INFO:tasks.rados.rados.0.smithi165.stdout:232: got expected ENOENT (src dne) 2024-09-16T04:42:21.140 INFO:tasks.rados.rados.0.smithi165.stdout:238: seq_num 99 ranges {1818624=507904,2326528=606208,2932736=557056} 2024-09-16T04:42:21.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T04:42:21.181 INFO:tasks.rados.rados.0.smithi165.stdout:238: writing smithi16530038-49 from 1818624 to 2326528 tid 1 2024-09-16T04:42:21.190 INFO:tasks.rados.rados.0.smithi165.stdout:238: writing smithi16530038-49 from 2326528 to 2932736 tid 2 2024-09-16T04:42:21.195 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:21.191+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:21.197 INFO:tasks.rados.rados.0.smithi165.stdout:238: writing smithi16530038-49 from 2932736 to 3489792 tid 3 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:239: copy_from oid 2 from oid 22 current snap is 16 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:232: finishing copy_from racing read to smithi16530038-41 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing rollback tid 0 to smithi16530038-42 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing rollback tid 1 to smithi16530038-42 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing rollback tid 2 to smithi16530038-42 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 112 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:234: finishing write tid 1 to smithi16530038-17 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:234: finishing write tid 2 to smithi16530038-17 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:234: finishing write tid 3 to smithi16530038-17 2024-09-16T04:42:21.198 INFO:tasks.rados.rados.0.smithi165.stdout:234: finishing write tid 4 to smithi16530038-17 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:234: finishing write tid 5 to smithi16530038-17 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:230: done (7 left) 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:232: done (6 left) 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:240: delete oid 27 current snap is 16 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:233: finishing rollback tid 0 to smithi16530038-16 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:233: finishing rollback tid 1 to smithi16530038-16 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:233: finishing rollback tid 2 to smithi16530038-16 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 48 (ObjNum 75 snap 6 seq_num 75) dirty exists 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:234: finishing write tid 6 to smithi16530038-17 2024-09-16T04:42:21.199 INFO:tasks.rados.rados.0.smithi165.stdout:234: oid 17 updating version 0 to 52 2024-09-16T04:42:21.200 INFO:tasks.rados.rados.0.smithi165.stdout:234: oid 17 version 52 is already newer than 51 2024-09-16T04:42:21.200 INFO:tasks.rados.rados.0.smithi165.stdout:234: oid 17 updating version 52 to 55 2024-09-16T04:42:21.200 INFO:tasks.rados.rados.0.smithi165.stdout:234: oid 17 version 55 is already newer than 54 2024-09-16T04:42:21.200 INFO:tasks.rados.rados.0.smithi165.stdout:234: oid 17 version 55 is already newer than 53 2024-09-16T04:42:21.200 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 55 (ObjNum 98 snap 16 seq_num 98) dirty exists 2024-09-16T04:42:21.200 INFO:tasks.rados.rados.0.smithi165.stdout:234: left oid 17 (ObjNum 98 snap 16 seq_num 98) 2024-09-16T04:42:21.241 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:21.241 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:21.262 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:21.259+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:21.271 INFO:tasks.rados.rados.0.smithi165.stdout:239: finishing copy_from to smithi16530038-2 2024-09-16T04:42:21.271 INFO:tasks.rados.rados.0.smithi165.stdout:239: got expected ENOENT (src dne) 2024-09-16T04:42:21.272 INFO:tasks.rados.rados.0.smithi165.stdout:239: finishing copy_from racing read to smithi16530038-2 2024-09-16T04:42:21.280 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:21.280 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:21.280 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:21.280 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:21.280 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:21.283 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:21.283 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:21.294 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:42:21.305 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:21.305 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:21.328 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:21.329 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:21.359 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:21.355+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:21.374 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:21.460 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:21.460 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:21.461 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:21.455+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:21.470 INFO:tasks.rados.rados.0.smithi165.stdout:238: finishing write tid 1 to smithi16530038-49 2024-09-16T04:42:21.476 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:42:21.561 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:21.559+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:21.566 INFO:tasks.rados.rados.0.smithi165.stdout:238: finishing write tid 2 to smithi16530038-49 2024-09-16T04:42:21.567 INFO:tasks.rados.rados.0.smithi165.stdout:238: finishing write tid 3 to smithi16530038-49 2024-09-16T04:42:21.583 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:21.583 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:21.650 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:21.650 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:21.662 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:21.659+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:21.665 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:42:21.671 INFO:tasks.rados.rados.0.smithi165.stdout:238: finishing write tid 4 to smithi16530038-49 2024-09-16T04:42:21.701 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing rollback tid 0 to smithi16530038-45 2024-09-16T04:42:21.701 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing rollback tid 1 to smithi16530038-45 2024-09-16T04:42:21.702 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing rollback tid 2 to smithi16530038-45 2024-09-16T04:42:21.702 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 31 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T04:42:21.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:21.762 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:21.763 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:21.755+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:21.789 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:21.789 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:21.801 INFO:tasks.rados.rados.0.smithi165.stdout:233: done (6 left) 2024-09-16T04:42:21.801 INFO:tasks.rados.rados.0.smithi165.stdout:234: done (5 left) 2024-09-16T04:42:21.801 INFO:tasks.rados.rados.0.smithi165.stdout:235: done (4 left) 2024-09-16T04:42:21.801 INFO:tasks.rados.rados.0.smithi165.stdout:239: done (3 left) 2024-09-16T04:42:21.801 INFO:tasks.rados.rados.0.smithi165.stdout:240: done (2 left) 2024-09-16T04:42:21.801 INFO:tasks.rados.rados.0.smithi165.stdout:241: delete oid 38 current snap is 16 2024-09-16T04:42:21.803 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:42:21.863 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:21.859+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:21.889 INFO:tasks.rados.rados.0.smithi165.stdout:238: finishing write tid 5 to smithi16530038-49 2024-09-16T04:42:21.890 INFO:tasks.rados.rados.0.smithi165.stdout:238: oid 49 updating version 0 to 50 2024-09-16T04:42:21.890 INFO:tasks.rados.rados.0.smithi165.stdout:238: oid 49 updating version 50 to 53 2024-09-16T04:42:21.890 INFO:tasks.rados.rados.0.smithi165.stdout:238: oid 49 version 53 is already newer than 52 2024-09-16T04:42:21.890 INFO:tasks.rados.rados.0.smithi165.stdout:238: oid 49 version 53 is already newer than 51 2024-09-16T04:42:21.890 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 53 (ObjNum 99 snap 16 seq_num 99) dirty exists 2024-09-16T04:42:21.890 INFO:tasks.rados.rados.0.smithi165.stdout:238: left oid 49 (ObjNum 99 snap 16 seq_num 99) 2024-09-16T04:42:21.964 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:21.959+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:21.972 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:21.972 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:21.986 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:42:22.064 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:22.059+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:22.118 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:22.119 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:22.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:42:22.166 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:22.163+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:22.169 INFO:tasks.rados.rados.0.smithi165.stdout:238: done (2 left) 2024-09-16T04:42:22.169 INFO:tasks.rados.rados.0.smithi165.stdout:241: done (1 left) 2024-09-16T04:42:22.169 INFO:tasks.rados.rados.0.smithi165.stdout:242: setattr oid 48 current snap is 16 2024-09-16T04:42:22.176 INFO:tasks.rados.rados.0.smithi165.stdout:243: snap_remove snap 5 2024-09-16T04:42:22.251 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:22.251 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:22.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:42:22.267 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:22.263+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:22.325 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:243: done (2 left) 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 11 current snap is 16 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:221: finishing write tid 5 to smithi16530038-13 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:221: oid 13 updating version 0 to 45 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:221: oid 13 updating version 45 to 47 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:221: oid 13 version 47 is already newer than 44 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:221: oid 13 version 47 is already newer than 46 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 47 (ObjNum 93 snap 16 seq_num 93) dirty exists 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:221: left oid 13 (ObjNum 93 snap 16 seq_num 93) 2024-09-16T04:42:22.330 INFO:tasks.rados.rados.0.smithi165.stdout:244: seq_num 102 ranges {0=557056,557056=638976,1196032=524288} 2024-09-16T04:42:22.340 INFO:tasks.rados.rados.0.smithi165.stdout:244: writing smithi16530038-11 from 0 to 557056 tid 1 2024-09-16T04:42:22.351 INFO:tasks.rados.rados.0.smithi165.stdout:244: writing smithi16530038-11 from 557056 to 1196032 tid 2 2024-09-16T04:42:22.355 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T04:42:22.359 INFO:tasks.rados.rados.0.smithi165.stdout:244: writing smithi16530038-11 from 1196032 to 1720320 tid 3 2024-09-16T04:42:22.360 INFO:tasks.rados.rados.0.smithi165.stdout:221: done (2 left) 2024-09-16T04:42:22.360 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 49 current snap is 16 2024-09-16T04:42:22.360 INFO:tasks.rados.rados.0.smithi165.stdout:245: seq_num 103 ranges {3489792=573440,4063232=786432,4849664=376832} 2024-09-16T04:42:22.394 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:22.394 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:22.397 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:22.395+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:22.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:42:22.411 INFO:tasks.rados.rados.0.smithi165.stdout:245: writing smithi16530038-49 from 3489792 to 4063232 tid 1 2024-09-16T04:42:22.417 INFO:tasks.rados.rados.0.smithi165.stdout:245: writing smithi16530038-49 from 4063232 to 4849664 tid 2 2024-09-16T04:42:22.421 INFO:tasks.rados.rados.0.smithi165.stdout:245: writing smithi16530038-49 from 4849664 to 5226496 tid 3 2024-09-16T04:42:22.421 INFO:tasks.rados.rados.0.smithi165.stdout:246: copy_from oid 27 from oid 15 current snap is 16 2024-09-16T04:42:22.421 INFO:tasks.rados.rados.0.smithi165.stdout:247: snap_create 2024-09-16T04:42:22.468 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:22.463+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:22.494 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 124 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:42:22.537 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:22.537 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:22.551 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:42:22.598 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:22.595+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:22.670 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:22.667+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:22.673 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:22.673 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:22.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:42:22.771 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:22.767+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:22.825 INFO:tasks.rados.rados.0.smithi165.stdout:245: finishing write tid 1 to smithi16530038-49 2024-09-16T04:42:22.857 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:22.857 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:22.871 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:42:22.872 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:22.867+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:22.907 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 1 to smithi16530038-11 2024-09-16T04:42:22.925 INFO:tasks.rados.rados.0.smithi165.stdout:245: finishing write tid 2 to smithi16530038-49 2024-09-16T04:42:22.926 INFO:tasks.rados.rados.0.smithi165.stdout:245: finishing write tid 3 to smithi16530038-49 2024-09-16T04:42:22.926 INFO:tasks.rados.rados.0.smithi165.stdout:245: finishing write tid 4 to smithi16530038-49 2024-09-16T04:42:22.929 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:22.923+0000 7f2552128640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 7 after trimming was started 2024-09-16T04:42:22.973 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:22.967+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:22.992 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:22.992 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:23.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:42:23.034 INFO:tasks.rados.rados.0.smithi165.stdout:246: finishing copy_from racing read to smithi16530038-27 2024-09-16T04:42:23.074 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:23.067+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:23.130 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:23.130 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:23.131 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 2 to smithi16530038-11 2024-09-16T04:42:23.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:42:23.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:42:23.190 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:23.187+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:23.234 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 3 to smithi16530038-11 2024-09-16T04:42:23.237 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 4 to smithi16530038-11 2024-09-16T04:42:23.264 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:23.265 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:23.276 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:23.271+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:23.279 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:42:23.331 INFO:tasks.rados.rados.0.smithi165.stdout:245: finishing write tid 5 to smithi16530038-49 2024-09-16T04:42:23.331 INFO:tasks.rados.rados.0.smithi165.stdout:245: oid 49 updating version 0 to 61 2024-09-16T04:42:23.332 INFO:tasks.rados.rados.0.smithi165.stdout:245: oid 49 version 61 is already newer than 60 2024-09-16T04:42:23.332 INFO:tasks.rados.rados.0.smithi165.stdout:245: oid 49 version 61 is already newer than 59 2024-09-16T04:42:23.332 INFO:tasks.rados.rados.0.smithi165.stdout:245: oid 49 version 61 is already newer than 58 2024-09-16T04:42:23.332 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 61 (ObjNum 103 snap 16 seq_num 103) dirty exists 2024-09-16T04:42:23.332 INFO:tasks.rados.rados.0.smithi165.stdout:245: left oid 49 (ObjNum 103 snap 16 seq_num 103) 2024-09-16T04:42:23.334 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T04:42:23.338 INFO:tasks.rados.rados.0.smithi165.stdout:242: done (4 left) 2024-09-16T04:42:23.338 INFO:tasks.rados.rados.0.smithi165.stdout:245: done (3 left) 2024-09-16T04:42:23.338 INFO:tasks.rados.rados.0.smithi165.stdout:247: done (2 left) 2024-09-16T04:42:23.338 INFO:tasks.rados.rados.0.smithi165.stdout:248: snap_create 2024-09-16T04:42:23.361 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T04:42:23.410 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:23.407+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:23.436 INFO:tasks.rados.rados.0.smithi165.stdout:246: finishing copy_from to smithi16530038-27 2024-09-16T04:42:23.436 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:42:23.447 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:23.447 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:23.454 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 5 to smithi16530038-11 2024-09-16T04:42:23.454 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 11 updating version 0 to 110 2024-09-16T04:42:23.454 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 11 updating version 110 to 112 2024-09-16T04:42:23.454 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 11 version 112 is already newer than 111 2024-09-16T04:42:23.454 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 11 version 112 is already newer than 109 2024-09-16T04:42:23.455 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 112 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-16T04:42:23.455 INFO:tasks.rados.rados.0.smithi165.stdout:244: left oid 11 (ObjNum 102 snap 16 seq_num 102) 2024-09-16T04:42:23.462 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:42:23.477 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:23.471+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:23.577 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:23.575+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:23.582 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:23.583 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:23.597 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:42:23.678 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:23.675+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:23.714 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:23.715 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:23.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:42:23.780 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:23.775+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:23.797 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:23.863 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:23.863 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:23.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:42:23.881 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:23.875+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:23.959 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:23.959 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:23.980 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:23.981 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:23.975+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:24.001 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:24.002 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:24.002 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:24.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:42:24.020 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:24.020 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:24.062 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:24.063 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:24.083 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:24.079+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:24.101 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:24.101 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:24.130 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:24.130 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:122 3.as0 3:5619390f:::smithi16530038-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:24.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.460268579999997, 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51059284400000005, 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678088+0000", 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.12e-06 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678098+0000", 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0000000000000001e-05 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678119+0000", 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1676e-05 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678135+0000", 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5526999999999999e-05 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682759+0000", 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046242649999999998 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682887+0000", 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012783599999999999 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.683968+0000", 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010808090000000001 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.784795+0000", 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10082761899999999 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188432+0000", 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40363656399999998 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188678+0000", 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024642799999999998 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:123 3.as0 3:5619390f:::smithi16530038-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.460137244000002, 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91283640799999999, 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.148 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 123 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678219+0000", 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4030000000000001e-06 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678231+0000", 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678267+0000", 2024-09-16T04:42:24.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5719000000000001e-05 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678291+0000", 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4006e-05 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684378+0000", 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060877320000000002 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:24.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684496+0000", 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118076 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188804+0000", 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50430794400000001 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.189032+0000", 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000227235 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.591053+0000", 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402021666 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.154 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:133 3.as0 3:5f4281cc:::smithi16530038-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.254949058999998, 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50755930599999999, 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883408+0000", 2024-09-16T04:42:24.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8669999999999998e-06 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884235+0000", 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082694699999999997 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884270+0000", 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4690000000000002e-05 2024-09-16T04:42:24.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884286+0000", 2024-09-16T04:42:24.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6082999999999998e-05 2024-09-16T04:42:24.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884441+0000", 2024-09-16T04:42:24.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155852 2024-09-16T04:42:24.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884838+0000", 2024-09-16T04:42:24.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000396494 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.886865+0000", 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020266709999999999 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965465+0000", 2024-09-16T04:42:24.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078600544999999994 2024-09-16T04:42:24.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390790+0000", 2024-09-16T04:42:24.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42532448499999997 2024-09-16T04:42:24.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390964+0000", 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017467200000000001 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:134 3.as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.254037568999998, 2024-09-16T04:42:24.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50704782699999995, 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:24.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884318+0000", 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884326+0000", 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.32e-06 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884339+0000", 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.274e-05 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884348+0000", 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0229999999999998e-06 2024-09-16T04:42:24.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888432+0000", 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040844899999999996 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888564+0000", 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131419 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889204+0000", 2024-09-16T04:42:24.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064011899999999997 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965469+0000", 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076265854999999994 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391186+0000", 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.425716382 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391364+0000", 2024-09-16T04:42:24.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017849599999999999 2024-09-16T04:42:24.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.168 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:135 3.as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.253943473, 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0075776860000001, 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884412+0000", 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3399999999999997e-07 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884416+0000", 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3059999999999999e-06 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884428+0000", 2024-09-16T04:42:24.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2500000000000001e-05 2024-09-16T04:42:24.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884439+0000", 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0662999999999999e-05 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889599+0000", 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051597179999999998 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:24.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889722+0000", 2024-09-16T04:42:24.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012330300000000001 2024-09-16T04:42:24.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392670+0000", 2024-09-16T04:42:24.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50294774399999997 2024-09-16T04:42:24.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392905+0000", 2024-09-16T04:42:24.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023530799999999999 2024-09-16T04:42:24.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.891988+0000", 2024-09-16T04:42:24.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49908321 2024-09-16T04:42:24.176 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.176 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.177 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:171 3.as0 3:5619390f:::smithi16530038-41:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.177 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896540+0000", 2024-09-16T04:42:24.177 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.241814431999998, 2024-09-16T04:42:24.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40213990599999999, 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896540+0000", 2024-09-16T04:42:24.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896540+0000", 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896542+0000", 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.554e-06 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896552+0000", 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.01e-05 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896583+0000", 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0423e-05 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896598+0000", 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5043e-05 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896717+0000", 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118836 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896906+0000", 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018926199999999999 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.897736+0000", 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082986699999999997 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.899917+0000", 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021813000000000002 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.298570+0000", 2024-09-16T04:42:24.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39865285499999997 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.298680+0000", 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109666 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.184 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:176 3.as0 3:5619390f:::smithi16530038-41:head [copy-from ver 10] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.063656694999999, 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71635610400000005, 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:24.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074701+0000", 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9500000000000003e-06 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074716+0000", 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4523e-05 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074742+0000", 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5795999999999998e-05 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074764+0000", 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2646999999999998e-05 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074937+0000", 2024-09-16T04:42:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017243599999999999 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074957+0000", 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9873000000000001e-05 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178498+0000", 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103540929 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178630+0000", 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132399 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.493151+0000", 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.3145212 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.597069+0000", 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103917643 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.790921+0000", 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19385244300000001 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791054+0000", 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013226500000000001 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:177 3.as0 3:5619390f:::smithi16530038-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.063537162000003, 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42059460999999998, 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.192 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:24.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074819+0000", 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074828+0000", 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7399999999999993e-06 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074850+0000", 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2802999999999998e-05 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074873+0000", 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2180000000000001e-05 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075093+0000", 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000220645 2024-09-16T04:42:24.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075109+0000", 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5823000000000001e-05 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178898+0000", 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10378908100000001 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178962+0000", 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489000000000001e-05 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495097+0000", 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31613566599999998 2024-09-16T04:42:24.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495230+0000", 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133039 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495412+0000", 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018130800000000001 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.198 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:194 3.as0 3:5f4281cc:::smithi16530038-45:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.038378129999998, 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59800688199999996, 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 194 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:24.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099979+0000", 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6400000000000001e-06 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099990+0000", 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.15e-05 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100022+0000", 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1569999999999998e-05 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100039+0000", 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7486000000000002e-05 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179005+0000", 2024-09-16T04:42:24.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078965628999999996 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179189+0000", 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184155 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.486772+0000", 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30758345500000001 2024-09-16T04:42:24.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495583+0000", 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088103490000000003 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697796+0000", 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20221325300000001 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697983+0000", 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018684499999999999 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:276 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.039574645999998, 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71405619499999995, 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.205 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 276 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098783+0000", 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098798+0000", 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5213e-05 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098845+0000", 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6853e-05 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098872+0000", 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6622999999999999e-05 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099054+0000", 2024-09-16T04:42:24.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018249499999999999 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099075+0000", 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.065e-05 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207663+0000", 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858786400000001 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207828+0000", 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016534199999999999 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812836+0000", 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60500746100000002 2024-09-16T04:42:24.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:277 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.039426794000001, 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81467523799999997, 2024-09-16T04:42:24.211 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 277 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098929+0000", 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4670000000000001e-06 2024-09-16T04:42:24.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098938+0000", 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098964+0000", 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5979999999999999e-05 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098980+0000", 2024-09-16T04:42:24.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5917e-05 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099168+0000", 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000188228 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099176+0000", 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.307912+0000", 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20873598700000001 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.308068+0000", 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156448 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913603+0000", 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60553407699999995 2024-09-16T04:42:24.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:278 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.039350683999999, 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81515018400000006, 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 278 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099005+0000", 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-06 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099012+0000", 2024-09-16T04:42:24.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9e-06 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099031+0000", 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9303000000000001e-05 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099046+0000", 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5003e-05 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099213+0000", 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000166773 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099218+0000", 2024-09-16T04:42:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408464+0000", 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30924607700000001 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408612+0000", 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014793900000000001 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.914154+0000", 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50554194200000002 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.309194+0000", 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.829160244000001, 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40233833499999999, 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309194+0000", 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309194+0000", 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309196+0000", 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1260000000000001e-06 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309232+0000", 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6443999999999999e-05 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309253+0000", 2024-09-16T04:42:24.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0599999999999999e-05 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.309265+0000", 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2099999999999999e-05 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.711464+0000", 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402199313 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.711532+0000", 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7751999999999998e-05 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.as0 43/32 ECSubRead(tid=39, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.729433855, 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40299000499999998, 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.228 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:24.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408986+0000", 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409043+0000", 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6589999999999999e-05 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811767+0000", 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40272446200000001 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811910+0000", 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014330300000000001 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=37, attrs_read=0))", 2024-09-16T04:42:24.232 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.729034534, 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.403543547, 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409389+0000", 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9126000000000002e-05 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409407+0000", 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8097e-05 2024-09-16T04:42:24.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812374+0000", 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40296689699999999 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812863+0000", 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048942699999999996 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.728674077000001, 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40266660700000001, 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.237 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409681+0000", 2024-09-16T04:42:24.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2699999999999998e-07 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409710+0000", 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9036000000000001e-05 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409721+0000", 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1187e-05 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409729+0000", 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7260000000000002e-06 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812298+0000", 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40256918800000002 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812347+0000", 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8542999999999999e-05 2024-09-16T04:42:24.240 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.728597978, 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402371019, 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.241 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409757+0000", 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409802+0000", 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4969999999999998e-05 2024-09-16T04:42:24.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409813+0000", 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1127e-05 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409820+0000", 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4499999999999998e-06 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812023+0000", 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40220215599999998 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812127+0000", 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104549 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:42:24.245 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.728431299, 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40309429400000002, 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:24.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409924+0000", 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409963+0000", 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8753e-05 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409981+0000", 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8737e-05 2024-09-16T04:42:24.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409991+0000", 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3889999999999994e-06 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812922+0000", 2024-09-16T04:42:24.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40293080100000001 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.813017+0000", 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5457000000000004e-05 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.410028+0000", 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.728325763000001, 2024-09-16T04:42:24.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40223944, 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410028+0000", 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410028+0000", 2024-09-16T04:42:24.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410029+0000", 2024-09-16T04:42:24.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410061+0000", 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2246000000000003e-05 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.410076+0000", 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.504e-05 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.510190+0000", 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100113808 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812211+0000", 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30202088300000002 2024-09-16T04:42:24.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812268+0000", 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6765999999999998e-05 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:24.255 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.627351032, 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40262853199999998, 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:24.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511049+0000", 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5912999999999999e-05 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511068+0000", 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.855e-05 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913200+0000", 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40213229900000003 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913632+0000", 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043176999999999999 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.259 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.260 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.260 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.260 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:24.261 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:24.262 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:24.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:42:24.264 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:24.264 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:24.255+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:24.284 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:24.284 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:24.305 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:24.305 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:24.321 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:24.338 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:42:24.338 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T04:42:24.343 INFO:tasks.rados.rados.0.smithi165.stdout:244: done (2 left) 2024-09-16T04:42:24.343 INFO:tasks.rados.rados.0.smithi165.stdout:246: done (1 left) 2024-09-16T04:42:24.344 INFO:tasks.rados.rados.0.smithi165.stdout:248: done (0 left) 2024-09-16T04:42:24.344 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 17 current snap is 18 2024-09-16T04:42:24.344 INFO:tasks.rados.rados.0.smithi165.stdout:249: seq_num 104 ranges {1654784=589824,2244608=540672,2785280=622592} 2024-09-16T04:42:24.344 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:24.345 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:24.360 INFO:tasks.rados.rados.0.smithi165.stdout:249: writing smithi16530038-17 from 1654784 to 2244608 tid 1 2024-09-16T04:42:24.362 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:24.359+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:24.364 INFO:tasks.rados.rados.0.smithi165.stdout:249: writing smithi16530038-17 from 2244608 to 2785280 tid 2 2024-09-16T04:42:24.366 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:24.367 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:24.368 INFO:tasks.rados.rados.0.smithi165.stdout:249: writing smithi16530038-17 from 2785280 to 3407872 tid 3 2024-09-16T04:42:24.369 INFO:tasks.rados.rados.0.smithi165.stdout:250: rollback oid 8 current snap is 18 2024-09-16T04:42:24.369 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 15 2024-09-16T04:42:24.369 INFO:tasks.rados.rados.0.smithi165.stdout:251: rmattr oid 26 current snap is 18 2024-09-16T04:42:24.369 INFO:tasks.rados.rados.0.smithi165.stdout:252: delete oid 2 current snap is 18 2024-09-16T04:42:24.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T04:42:24.397 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:622 3.ds0 3.c6b9d6ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:24.369005+0000", 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.028119603999999999, 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028143741999999999, 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "op must wait for map", 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 622 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369005+0000", 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369005+0000", 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369007+0000", 2024-09-16T04:42:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.593e-06 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369017+0000", 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9669999999999996e-06 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369043+0000", 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5434000000000001e-05 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369053+0000", 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0817e-05 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369149+0000", 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5153000000000002e-05 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:24.369158+0000", 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4150000000000005e-06 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T04:42:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-16T04:42:24.402 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:24.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:24.463 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:24.459+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:24.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:24.503 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 58 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:42:24.563 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:24.559+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:24.583 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:24.584 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:24.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:24.664 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:24.659+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:251: done (3 left) 2024-09-16T04:42:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:252: done (2 left) 2024-09-16T04:42:24.680 INFO:tasks.rados.rados.0.smithi165.stdout:253: rmattr oid 24 current snap is 18 2024-09-16T04:42:24.680 INFO:tasks.rados.rados.0.smithi165.stdout:253: done (2 left) 2024-09-16T04:42:24.680 INFO:tasks.rados.rados.0.smithi165.stdout:254: snap_create 2024-09-16T04:42:24.717 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:24.717 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:24.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:142 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.538951875000002, 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98845964500000005, 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 142 2024-09-16T04:42:24.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175562+0000", 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6839999999999998e-06 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175978+0000", 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041531000000000001 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176014+0000", 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5913e-05 2024-09-16T04:42:24.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176030+0000", 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5872999999999999e-05 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658368+0000", 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.48233846800000002 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658643+0000", 2024-09-16T04:42:24.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000274651 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.659966+0000", 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013230399999999999 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665031+0000", 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050652290000000001 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.163900+0000", 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.498868961 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164018+0000", 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118516 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:143 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.723 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.532473512999999, 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.982808864, 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182041+0000", 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5099999999999999e-06 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182677+0000", 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063672200000000005 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182720+0000", 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2339999999999998e-05 2024-09-16T04:42:24.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182742+0000", 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1929999999999998e-05 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661153+0000", 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.478411009 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661298+0000", 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145659 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.662610+0000", 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001311967 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766105+0000", 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10349428300000001 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164694+0000", 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39858998099999998 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164846+0000", 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015146300000000001 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.729 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:144 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.531733692000003, 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98240055400000004, 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.730 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 144 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182779+0000", 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182787+0000", 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.977e-06 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182810+0000", 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3543000000000001e-05 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182839+0000", 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9e-05 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663652+0000", 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.480813243 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663795+0000", 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143108 2024-09-16T04:42:24.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.664540+0000", 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074409499999999996 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766192+0000", 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10165231600000001 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165013+0000", 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39882149300000003 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165177+0000", 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164146 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:145 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.735 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.531648828999998, 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.385587095, 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182863+0000", 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3039999999999999e-06 2024-09-16T04:42:24.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182868+0000", 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182891+0000", 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2362999999999999e-05 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182911+0000", 2024-09-16T04:42:24.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9813000000000002e-05 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665100+0000", 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.482189849 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665304+0000", 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020365800000000001 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264646+0000", 2024-09-16T04:42:24.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59934227699999998 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264794+0000", 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147412 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.568449+0000", 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30365514599999999 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.740 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:1 3.ds0 3:bc30766e:::smithi16530038-11:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.502119794999999, 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0541568889999999, 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:24.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212394+0000", 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9000000000000002e-06 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212406+0000", 2024-09-16T04:42:24.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2289999999999999e-05 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212428+0000", 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2410000000000001e-05 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212462+0000", 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3708999999999999e-05 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765550+0000", 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55308802099999999 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765692+0000", 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014137199999999999 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.266548+0000", 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50085618700000001 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.744 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:172 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.415307859000002, 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57524237600000006, 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.745 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 172 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299206+0000", 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6859999999999999e-06 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299276+0000", 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9592999999999996e-05 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299323+0000", 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6749999999999998e-05 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299347+0000", 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4416000000000001e-05 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469676+0000", 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.170329287 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469835+0000", 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159006 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.874445+0000", 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40460963799999999 2024-09-16T04:42:24.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:173 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.415123770999998, 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57613007299999996, 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299389+0000", 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7099999999999999e-06 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299399+0000", 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0866000000000001e-05 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299427+0000", 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7117000000000001e-05 2024-09-16T04:42:24.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299488+0000", 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1186000000000007e-05 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470324+0000", 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17083640999999999 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470484+0000", 2024-09-16T04:42:24.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015948199999999999 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.875517+0000", 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40503330199999998 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:174 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.414994991999997, 2024-09-16T04:42:24.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67607313300000005, 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299517+0000", 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.393e-06 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299526+0000", 2024-09-16T04:42:24.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8230000000000001e-06 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299553+0000", 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6889999999999998e-05 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299573+0000", 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9944000000000001e-05 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471098+0000", 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.171524865 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471221+0000", 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012369900000000001 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.975589+0000", 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50436751899999999 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.758 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:282 3.ds0 3:b100bb67:::smithi16530038-42:head [copy-from ver 3] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.615613216, 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62041838900000001, 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.759 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 282 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098900+0000", 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6630000000000002e-06 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098911+0000", 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0447e-05 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098945+0000", 2024-09-16T04:42:24.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4418999999999997e-05 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098960+0000", 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5119999999999999e-05 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099123+0000", 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016297600000000001 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:24.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099133+0000", 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0125999999999999e-05 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.303007+0000", 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20387381399999999 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.303177+0000", 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016986900000000001 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.615144+0000", 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31196737099999999 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.620953+0000", 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058083869999999999 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.719092+0000", 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098139689000000002 2024-09-16T04:42:24.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.719316+0000", 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000223508 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:283 3.ds0 3:b100bb67:::smithi16530038-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.099001+0000", 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.615509664000001, 2024-09-16T04:42:24.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51969615000000002, 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 283 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099001+0000", 2024-09-16T04:42:24.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099001+0000", 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.584e-06 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099011+0000", 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2630000000000003e-06 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099033+0000", 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2427000000000001e-05 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099049+0000", 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6033e-05 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099243+0000", 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019395799999999999 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099260+0000", 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6900000000000001e-05 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.403620+0000", 2024-09-16T04:42:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30435966399999997 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.403709+0000", 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9553000000000001e-05 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.618382+0000", 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.214672997 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.618533+0000", 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015042300000000001 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.618697+0000", 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016434800000000001 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:375 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~0] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:59.180661+0000", 2024-09-16T04:42:24.771 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.533849159999999, 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49449954699999998, 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 375 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180661+0000", 2024-09-16T04:42:24.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180661+0000", 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180662+0000", 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1299999999999998e-07 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180668+0000", 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5969999999999999e-06 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180681+0000", 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2826000000000001e-05 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180690+0000", 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7269999999999996e-06 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180860+0000", 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000170185 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.180869+0000", 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1260000000000004e-06 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.389145+0000", 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20827663699999999 2024-09-16T04:42:24.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.389352+0000", 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020640900000000001 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.390495+0000", 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001143018 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.592768+0000", 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20227287499999999 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.675114+0000", 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.082346266000000001 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.675161+0000", 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6968000000000002e-05 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:466 3.ds0 3:b6ec1d3e:::smithi16530038-48:a [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:08.304297+0000", 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.410213898999999, 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50756305899999998, 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 466 2024-09-16T04:42:24.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.304297+0000", 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.304297+0000", 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.304297+0000", 2024-09-16T04:42:24.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.304303+0000", 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.947e-06 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.304313+0000", 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.304319+0000", 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9680000000000003e-06 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.608496+0000", 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.304177064 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.608646+0000", 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015006000000000001 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:08.811860+0000", 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20321389400000001 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:513 3.ds0 3:b56b9d63:::smithi16530038-2:head [append 0~720896] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.4861354490000007, 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112358499999997, 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.782 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 513 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228379+0000", 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5269999999999999e-06 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.229180+0000", 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080095700000000004 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.229216+0000", 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6303000000000001e-05 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329324+0000", 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100107881 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329465+0000", 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140796 2024-09-16T04:42:24.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329663+0000", 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019788499999999999 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.332137+0000", 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024749519999999999 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437411+0000", 2024-09-16T04:42:24.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105273296 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839350+0000", 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40193957600000002 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839499+0000", 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000148412 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:514 3.ds0 3:b56b9d63:::smithi16530038-2:head [append 0~442368] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.329378+0000", 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3851323949999994, 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51044876699999997, 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 514 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.788 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329378+0000", 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329378+0000", 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329380+0000", 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329910+0000", 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00052939999999999997 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329934+0000", 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3853e-05 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329949+0000", 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5167000000000001e-05 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.333951+0000", 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040021029999999999 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.334165+0000", 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021436200000000001 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.336388+0000", 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022231299999999998 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437417+0000", 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101028405 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839745+0000", 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402328252 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839827+0000", 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2004999999999998e-05 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:515 3.ds0 3:b56b9d63:::smithi16530038-2:head [append 0~327680] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:24.793 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.329977+0000", 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3845337989999997, 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51048841099999998, 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 515 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329977+0000", 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329977+0000", 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329978+0000", 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1599999999999999e-06 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330336+0000", 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035811099999999999 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330351+0000", 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4473e-05 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330360+0000", 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2329999999999995e-06 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.337819+0000", 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074591509999999998 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.338063+0000", 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000243761 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.340407+0000", 2024-09-16T04:42:24.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002344793 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437420+0000", 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097012878999999996 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.840314+0000", 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40289380499999999 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.840465+0000", 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015104499999999999 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:516 3.ds0 3:b56b9d63:::smithi16530038-2:head [setxattr _header (56) in=7b] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.330375+0000", 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3841354419999998, 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51032915400000001, 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.799 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 516 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330375+0000", 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330375+0000", 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330376+0000", 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2699999999999998e-07 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330381+0000", 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3129999999999998e-06 2024-09-16T04:42:24.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330396+0000", 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4266e-05 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330404+0000", 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1000000000000004e-06 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.433965+0000", 2024-09-16T04:42:24.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10356122299999999 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434144+0000", 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017851499999999999 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.435030+0000", 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00088694100000000003 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437580+0000", 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025490909999999999 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.840643+0000", 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40306319600000001 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.840704+0000", 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1581999999999994e-05 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:517 3.ds0 3:b56b9d63:::smithi16530038-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3840955229999992, 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71224451200000005, 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.805 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 517 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330416+0000", 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330420+0000", 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.557e-06 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330431+0000", 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1223000000000001e-05 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330438+0000", 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7170000000000008e-06 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.841180+0000", 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51074163900000003 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.841322+0000", 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141433 2024-09-16T04:42:24.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.042660+0000", 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20133805299999999 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:568 3.ds0 3:b100bb67:::smithi16530038-42:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.8129973619999999, 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60988380600000003, 2024-09-16T04:42:24.809 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 568 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901535+0000", 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1696999999999999e-05 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901549+0000", 2024-09-16T04:42:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.364e-05 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901580+0000", 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1335999999999999e-05 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901639+0000", 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8776000000000001e-05 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.300597+0000", 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.398958549 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.300848+0000", 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025097800000000001 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.302460+0000", 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016119750000000001 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.306368+0000", 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039082070000000003 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.511231+0000", 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.204862715 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.511397+0000", 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000165933 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:569 3.ds0 3:b100bb67:::smithi16530038-42:head [rollback 11] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.8127610509999998, 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80813885399999996, 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 569 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901751+0000", 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6670000000000001e-06 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901804+0000", 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2306999999999999e-05 2024-09-16T04:42:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901826+0000", 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2186000000000001e-05 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901845+0000", 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8802999999999999e-05 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.409423+0000", 2024-09-16T04:42:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.507578164 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.409660+0000", 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023717500000000001 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.410803+0000", 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011429249999999999 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.611780+0000", 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20097747999999999 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.709760+0000", 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097980190999999994 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.709888+0000", 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012795600000000001 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:570 3.ds0 3:b100bb67:::smithi16530038-42:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.812636715, 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81140166199999997, 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 570 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901875+0000", 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.483e-06 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901883+0000", 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.357e-06 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901901+0000", 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8593e-05 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901917+0000", 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.556e-05 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.509597+0000", 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60768048500000005 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.509774+0000", 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000176909 2024-09-16T04:42:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.510566+0000", 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079207799999999999 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.611782+0000", 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101216059 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.713154+0000", 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10137199199999999 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.713276+0000", 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121146 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:24.826 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:24.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:24.828 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:24.823+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:24.914 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing rollback tid 0 to smithi16530038-8 2024-09-16T04:42:24.914 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing rollback tid 1 to smithi16530038-8 2024-09-16T04:42:24.915 INFO:tasks.rados.rados.0.smithi165.stdout:249: finishing write tid 1 to smithi16530038-17 2024-09-16T04:42:24.915 INFO:tasks.rados.rados.0.smithi165.stdout:249: finishing write tid 2 to smithi16530038-17 2024-09-16T04:42:24.915 INFO:tasks.rados.rados.0.smithi165.stdout:249: finishing write tid 3 to smithi16530038-17 2024-09-16T04:42:24.928 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:24.923+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:24.959 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:24.959 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:24.959 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:24.960 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:24.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:25.066 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:25.063+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:25.073 INFO:tasks.rados.rados.0.smithi165.stdout:249: finishing write tid 4 to smithi16530038-17 2024-09-16T04:42:25.131 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:25.127+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:25.136 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T04:42:25.140 INFO:tasks.rados.rados.0.smithi165.stdout:254: done (2 left) 2024-09-16T04:42:25.140 INFO:tasks.rados.rados.0.smithi165.stdout:255: copy_from oid 29 from oid 39 current snap is 19 2024-09-16T04:42:25.140 INFO:tasks.rados.rados.0.smithi165.stdout:256: read oid 36 snap -1 2024-09-16T04:42:25.140 INFO:tasks.rados.rados.0.smithi165.stdout:256: expect deleted 2024-09-16T04:42:25.140 INFO:tasks.rados.rados.0.smithi165.stdout:257: snap_create 2024-09-16T04:42:25.144 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:25.144 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:25.144 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:25.144 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:25.144 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:25.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:25.165 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T04:42:25.187 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing rollback tid 2 to smithi16530038-8 2024-09-16T04:42:25.187 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 53 (ObjNum 74 snap 5 seq_num 74) dirty exists 2024-09-16T04:42:25.231 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:25.227+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:154 3.fs0 3:f6276c08:::smithi16530038-20:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.09961414, 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2626387860000001, 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 154 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210502+0000", 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9139999999999999e-06 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210519+0000", 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6969999999999998e-05 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210555+0000", 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6099000000000001e-05 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210573+0000", 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7770000000000001e-05 2024-09-16T04:42:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210738+0000", 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016486499999999999 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210962+0000", 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022410600000000001 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.269739+0000", 2024-09-16T04:42:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.058777538 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272567+0000", 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028276550000000001 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.472907+0000", 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20033962699999999 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.473136+0000", 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000229242 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:155 3.fs0 3:f6276c08:::smithi16530038-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.099498130999997, 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.061292586, 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 155 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210614+0000", 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7000000000000003e-07 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210619+0000", 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210636+0000", 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7436999999999999e-05 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210645+0000", 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.816e-06 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211299+0000", 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065386899999999996 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211385+0000", 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5852000000000004e-05 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271332+0000", 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.05994666 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271584+0000", 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025255399999999999 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271906+0000", 2024-09-16T04:42:25.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032151100000000002 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:169 3.cs0 3:34145aa0:::smithi16530038-26:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.413434514000002, 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87942165800000005, 2024-09-16T04:42:25.327 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 169 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896679+0000", 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3070000000000001e-06 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896692+0000", 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2785999999999999e-05 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896719+0000", 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7540000000000001e-05 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896735+0000", 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5897e-05 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998807+0000", 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102071833 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998930+0000", 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012282600000000001 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.572766+0000", 2024-09-16T04:42:25.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57383636000000005 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574732+0000", 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019661589999999999 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.775926+0000", 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20119392799999999 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.776098+0000", 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172022 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:170 3.cs0 3:34145aa0:::smithi16530038-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.413348593999999, 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67734535900000004, 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.334 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896764+0000", 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896768+0000", 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896780+0000", 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2003e-05 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896791+0000", 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0876000000000001e-05 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101155+0000", 2024-09-16T04:42:25.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20436343900000001 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101252+0000", 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7195999999999996e-05 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573831+0000", 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.47257916700000002 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573966+0000", 2024-09-16T04:42:25.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134816 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574108+0000", 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014204199999999999 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:207 3.fs0 3:f1175b03:::smithi16530038-29:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.209702944999997, 2024-09-16T04:42:25.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.69161456600000004, 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 207 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100411+0000", 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100423+0000", 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.202e-05 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100454+0000", 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0895999999999997e-05 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100469+0000", 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5614e-05 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174514+0000", 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.074044911000000005 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174748+0000", 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023420200000000001 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.687971+0000", 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51322260399999997 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690498+0000", 2024-09-16T04:42:25.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002526978 2024-09-16T04:42:25.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791786+0000", 2024-09-16T04:42:25.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10128820500000001 2024-09-16T04:42:25.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.792023+0000", 2024-09-16T04:42:25.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000236772 2024-09-16T04:42:25.346 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.347 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.347 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:208 3.fs0 3:f1175b03:::smithi16530038-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:25.347 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:25.348 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.209612599000003, 2024-09-16T04:42:25.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.58977269499999996, 2024-09-16T04:42:25.348 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.348 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.348 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.348 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.349 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208 2024-09-16T04:42:25.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.349 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:25.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:25.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100500+0000", 2024-09-16T04:42:25.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:25.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100507+0000", 2024-09-16T04:42:25.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T04:42:25.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100524+0000", 2024-09-16T04:42:25.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.607e-05 2024-09-16T04:42:25.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100532+0000", 2024-09-16T04:42:25.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8270000000000004e-06 2024-09-16T04:42:25.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274070+0000", 2024-09-16T04:42:25.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17353716399999999 2024-09-16T04:42:25.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:25.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274196+0000", 2024-09-16T04:42:25.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012600600000000001 2024-09-16T04:42:25.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689716+0000", 2024-09-16T04:42:25.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41552020499999998 2024-09-16T04:42:25.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689979+0000", 2024-09-16T04:42:25.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026336099999999999 2024-09-16T04:42:25.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690271+0000", 2024-09-16T04:42:25.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029222200000000002 2024-09-16T04:42:25.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.359 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.359 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.7.22:2 3.cs0 3:34145aa0:::smithi16530038-26:head [assert-version v13,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 30.990690001000001, 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61025086299999998, 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:25.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319424+0000", 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-16T04:42:25.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319433+0000", 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4239999999999999e-06 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319460+0000", 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6965999999999999e-05 2024-09-16T04:42:25.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319498+0000", 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7518999999999998e-05 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.421237+0000", 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101738922 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.421547+0000", 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000310871 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.518325+0000", 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096777742 2024-09-16T04:42:25.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.518478+0000", 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015335199999999999 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.929672+0000", 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41119360100000002 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.365 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:356 3.fs0 3:f6276c08:::smithi16530038-20:head [append 0~0] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.052576525999999, 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50700994099999996, 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 356 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257538+0000", 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3170000000000002e-06 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257552+0000", 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.381e-05 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257588+0000", 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6046e-05 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357861+0000", 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10027301499999999 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458138+0000", 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100277112 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458411+0000", 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027305100000000001 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.460377+0000", 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019655160000000001 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465044+0000", 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046678409999999998 2024-09-16T04:42:25.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.764419+0000", 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29937459100000002 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.764545+0000", 2024-09-16T04:42:25.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125642 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:357 3.fs0 3:f6276c08:::smithi16530038-20:head [rollback 3] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.952155104999999, 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60733727699999995, 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 357 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357959+0000", 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6630000000000002e-06 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357971+0000", 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2660000000000001e-05 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.358002+0000", 2024-09-16T04:42:25.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0793000000000001e-05 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458105+0000", 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100102386 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.462250+0000", 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004145213 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.462405+0000", 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015478599999999999 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463262+0000", 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085764999999999995 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465233+0000", 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019711049999999999 2024-09-16T04:42:25.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.965176+0000", 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.499942461 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.965293+0000", 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011756000000000001 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:358 3.fs0 3:f6276c08:::smithi16530038-20:head [append 0~0] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.851976053000001, 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60770323100000001, 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 358 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.379 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458137+0000", 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0229999999999999e-06 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458148+0000", 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0784e-05 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458166+0000", 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8116000000000001e-05 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458182+0000", 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5424e-05 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463813+0000", 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056313439999999999 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463964+0000", 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105300000000001 2024-09-16T04:42:25.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.464741+0000", 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077701400000000002 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465465+0000", 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072419100000000001 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.065719+0000", 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60025337300000003 2024-09-16T04:42:25.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.065838+0000", 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011990899999999999 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:376 3.cs0 3:376489e5:::smithi16530038-34:head [delete] snapc 6=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.0421601, 2024-09-16T04:42:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.52363831800000005, 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 376 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:25.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267954+0000", 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5339999999999998e-06 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267963+0000", 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6700000000000006e-06 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267992+0000", 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8646e-05 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268013+0000", 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0814000000000001e-05 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268212+0000", 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000199681 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268360+0000", 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014787599999999999 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.269584+0000", 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012233210000000001 2024-09-16T04:42:25.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.271521+0000", 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373459999999999 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.791438+0000", 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51991722100000004 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.791589+0000", 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151209 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:424 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~573440] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.075178979, 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.502816084, 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 424 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234936+0000", 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.456e-06 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.235611+0000", 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00067546800000000003 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.235647+0000", 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5466999999999998e-05 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335792+0000", 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10014563999999999 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335931+0000", 2024-09-16T04:42:25.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000138582 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336098+0000", 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167465 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.338085+0000", 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019867890000000001 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737213+0000", 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39912832100000001 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737613+0000", 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039929100000000002 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737748+0000", 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013560499999999999 2024-09-16T04:42:25.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:426 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~344064] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.973721597000001, 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90840981600000004, 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 426 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336391+0000", 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0699999999999996e-07 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336788+0000", 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039739000000000002 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336805+0000", 2024-09-16T04:42:25.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6583999999999999e-05 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336814+0000", 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4299999999999995e-06 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739098+0000", 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40228381099999999 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739254+0000", 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156506 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.740433+0000", 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011782450000000001 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943229+0000", 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20279692899999999 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244590+0000", 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30136072600000002 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244799+0000", 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020928799999999999 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.401 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:427 3.cs0 3:3dc0a373:::smithi16530038-10:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.97328053, 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90837145699999999, 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 427 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336832+0000", 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7400000000000002e-07 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336837+0000", 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9300000000000002e-06 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336857+0000", 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-05 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336890+0000", 2024-09-16T04:42:25.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2852000000000003e-05 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841600+0000", 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50471004399999997 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841814+0000", 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000214589 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.842536+0000", 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072138400000000004 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943303+0000", 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10076771499999999 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245048+0000", 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30174494299999999 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245202+0000", 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015384599999999999 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:428 3.cs0 3:3dc0a373:::smithi16530038-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.973208423999999, 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1109408999999999, 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.407 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 428 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336904+0000", 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336920+0000", 2024-09-16T04:42:25.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6067e-05 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336933+0000", 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336941+0000", 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4600000000000003e-06 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245817+0000", 2024-09-16T04:42:25.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90887578899999999 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.246090+0000", 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000273071 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.447844+0000", 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20175383 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:509 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~1458176,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:14.272093+0000", 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.038017798, 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41391541300000001, 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 509 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.412 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272093+0000", 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272093+0000", 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272097+0000", 2024-09-16T04:42:25.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0970000000000002e-06 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272110+0000", 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3536999999999999e-05 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272143+0000", 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2982999999999999e-05 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272160+0000", 2024-09-16T04:42:25.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6759999999999999e-05 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.372517+0000", 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10035675600000001 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.372528+0000", 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1083e-05 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.473768+0000", 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101240116 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.473944+0000", 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017610600000000001 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.686009+0000", 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21206497499999999 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.416 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:510 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~1458176,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:14.272190+0000", 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.037921442, 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41544610500000001, 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.417 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 510 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272190+0000", 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272190+0000", 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272191+0000", 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.017e-06 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272197+0000", 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2230000000000004e-06 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272213+0000", 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5950000000000001e-05 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272222+0000", 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7830000000000004e-06 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.472771+0000", 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20054923099999999 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.472783+0000", 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1507e-05 2024-09-16T04:42:25.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.474927+0000", 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021449210000000002 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.475141+0000", 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000213761 2024-09-16T04:42:25.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.687636+0000", 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.212494712 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:511 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~1458176,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.037876632, 2024-09-16T04:42:25.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.416826537, 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 511 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:25.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6300000000000004e-07 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272241+0000", 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272255+0000", 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3223e-05 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272263+0000", 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0199999999999994e-06 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.472829+0000", 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.200566153 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.472837+0000", 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2369999999999992e-06 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.475728+0000", 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028907859999999998 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.475882+0000", 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000154759 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.689061+0000", 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.213178642 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:519 3.fs0 3:f1175b03:::smithi16530038-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:25.427 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.076768491999999, 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83548977300000005, 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 519 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233344+0000", 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233352+0000", 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5100000000000001e-06 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233381+0000", 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8887e-05 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233398+0000", 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7427000000000001e-05 2024-09-16T04:42:25.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233732+0000", 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033334700000000001 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233749+0000", 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7612999999999999e-05 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236234+0000", 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024850919999999999 2024-09-16T04:42:25.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236334+0000", 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100159 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068497+0000", 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83216263000000001 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068656+0000", 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015873499999999999 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068833+0000", 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017668299999999999 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:534 3.fs0 3:f6276c08:::smithi16530038-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:42:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.051375984, 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81072121699999999, 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 534 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:42:25.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258737+0000", 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.46e-06 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258746+0000", 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3670000000000003e-06 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258767+0000", 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1253000000000001e-05 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258785+0000", 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8046e-05 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.561788+0000", 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30300253199999999 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.562084+0000", 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029564400000000002 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.662783+0000", 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10069991 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.663047+0000", 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026331899999999998 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.069456+0000", 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40640968599999999 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.438 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:25.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:42:25.440 INFO:tasks.rados.rados.0.smithi165.stdout:249: finishing write tid 5 to smithi16530038-17 2024-09-16T04:42:25.441 INFO:tasks.rados.rados.0.smithi165.stdout:249: oid 17 updating version 0 to 70 2024-09-16T04:42:25.441 INFO:tasks.rados.rados.0.smithi165.stdout:249: oid 17 updating version 70 to 71 2024-09-16T04:42:25.441 INFO:tasks.rados.rados.0.smithi165.stdout:249: oid 17 version 71 is already newer than 69 2024-09-16T04:42:25.441 INFO:tasks.rados.rados.0.smithi165.stdout:249: oid 17 version 71 is already newer than 68 2024-09-16T04:42:25.441 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 71 (ObjNum 104 snap 18 seq_num 104) dirty exists 2024-09-16T04:42:25.441 INFO:tasks.rados.rados.0.smithi165.stdout:249: left oid 17 (ObjNum 104 snap 18 seq_num 104) 2024-09-16T04:42:25.441 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:25.435+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:25.491 INFO:tasks.rados.rados.0.smithi165.stdout:255: finishing copy_from racing read to smithi16530038-29 2024-09-16T04:42:25.544 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:25.535+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:25.614 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:25.615 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:25.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:42:25.642 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:25.639+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:25.743 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:25.739+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:25.787 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:25.787 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:25.787 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:25.787 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:25.788 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:25.791 INFO:tasks.rados.rados.0.smithi165.stdout:255: finishing copy_from to smithi16530038-29 2024-09-16T04:42:25.791 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T04:42:25.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:42:25.844 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:25.839+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:25.925 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:25.925 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:25.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:25.925 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:3 3.5s0 3:a54da884:::smithi16530038-14:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.745607204999999, 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50899456499999995, 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:25.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176463+0000", 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176482+0000", 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8975999999999999e-05 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176519+0000", 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6916999999999999e-05 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.276624+0000", 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100105124 2024-09-16T04:42:25.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478009+0000", 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.201384806 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478333+0000", 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000324381 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.685454+0000", 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20712119400000001 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:359 3.5s0 3:a54da884:::smithi16530038-14:head [read 0~229376,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257726+0000", 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.664340878000001, 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30495125899999997, 2024-09-16T04:42:25.930 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 359 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257726+0000", 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257726+0000", 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257730+0000", 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6160000000000002e-06 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257749+0000", 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.942e-05 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257782+0000", 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2772999999999999e-05 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257798+0000", 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6626999999999999e-05 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.349728+0000", 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.091929364 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.349891+0000", 2024-09-16T04:42:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016355499999999999 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.562677+0000", 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.212785904 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.936 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:360 3.5s0 3:a54da884:::smithi16530038-14:head [read 0~229376,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:25.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257826+0000", 2024-09-16T04:42:25.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.664240728999999, 2024-09-16T04:42:25.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30567145899999998, 2024-09-16T04:42:25.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 360 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257826+0000", 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257826+0000", 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257827+0000", 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.47e-07 2024-09-16T04:42:25.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257833+0000", 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.246e-06 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257846+0000", 2024-09-16T04:42:25.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2787e-05 2024-09-16T04:42:25.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257855+0000", 2024-09-16T04:42:25.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0459999999999994e-06 2024-09-16T04:42:25.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.350510+0000", 2024-09-16T04:42:25.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092655111999999998 2024-09-16T04:42:25.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.350754+0000", 2024-09-16T04:42:25.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024403900000000001 2024-09-16T04:42:25.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.563498+0000", 2024-09-16T04:42:25.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21274328200000001 2024-09-16T04:42:25.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.943 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.943 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:361 3.5s0 3:a54da884:::smithi16530038-14:head [read 0~229376,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:25.944 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:25.944 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.664198815999999, 2024-09-16T04:42:25.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.306317012, 2024-09-16T04:42:25.944 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.944 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.944 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.944 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.945 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 361 2024-09-16T04:42:25.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.945 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257869+0000", 2024-09-16T04:42:25.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-16T04:42:25.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257874+0000", 2024-09-16T04:42:25.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.023e-06 2024-09-16T04:42:25.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257885+0000", 2024-09-16T04:42:25.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-16T04:42:25.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257892+0000", 2024-09-16T04:42:25.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5630000000000003e-06 2024-09-16T04:42:25.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.351201+0000", 2024-09-16T04:42:25.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093309317000000003 2024-09-16T04:42:25.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.351321+0000", 2024-09-16T04:42:25.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119459 2024-09-16T04:42:25.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.564185+0000", 2024-09-16T04:42:25.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21286419300000001 2024-09-16T04:42:25.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.952 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.952 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:392 3.5s0 3:a54da884:::smithi16530038-14:head [copy-from ver 8] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.779664147999998, 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.36993664399999998, 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.954 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 392 2024-09-16T04:42:25.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.956 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:25.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:25.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142404+0000", 2024-09-16T04:42:25.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.674e-06 2024-09-16T04:42:25.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142411+0000", 2024-09-16T04:42:25.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0400000000000004e-06 2024-09-16T04:42:25.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142438+0000", 2024-09-16T04:42:25.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6486e-05 2024-09-16T04:42:25.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142449+0000", 2024-09-16T04:42:25.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0835999999999999e-05 2024-09-16T04:42:25.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142539+0000", 2024-09-16T04:42:25.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9823999999999999e-05 2024-09-16T04:42:25.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142675+0000", 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013676899999999999 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.294617+0000", 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15194110199999999 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.308789+0000", 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014172736999999999 2024-09-16T04:42:25.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.512242+0000", 2024-09-16T04:42:25.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20345274799999999 2024-09-16T04:42:25.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.512339+0000", 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7428000000000003e-05 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.964 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:520 3.5s0 3:a54c6f35:::smithi16530038-38:head [append 0~0] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.232996+0000", 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.68907123, 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41309273600000002, 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.966 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 520 2024-09-16T04:42:25.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.966 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.232996+0000", 2024-09-16T04:42:25.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.232996+0000", 2024-09-16T04:42:25.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.232999+0000", 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8739999999999999e-06 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233010+0000", 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.099e-05 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233044+0000", 2024-09-16T04:42:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4391999999999999e-05 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233067+0000", 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3323e-05 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233303+0000", 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023559899999999999 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233625+0000", 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032232700000000002 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236680+0000", 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003055168 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545473+0000", 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30879239800000002 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.645946+0000", 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100472882 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646088+0000", 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142783 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:521 3.5s0 3:a54c6f35:::smithi16530038-38:head [rollback f] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.688943394000001, 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.514017481, 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.972 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 521 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233125+0000", 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.477e-06 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233131+0000", 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9660000000000001e-06 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233176+0000", 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5086000000000002e-05 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233194+0000", 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7706999999999998e-05 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.239157+0000", 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059635820000000003 2024-09-16T04:42:25.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.239414+0000", 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025656799999999999 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240993+0000", 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015795749999999999 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545483+0000", 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.304489921 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746960+0000", 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.201476977 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747141+0000", 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018062199999999999 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.977 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:522 3.5s0 3:a54c6f35:::smithi16530038-38:head [append 0~0] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.233288+0000", 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.688779412000001, 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51453239399999995, 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 522 2024-09-16T04:42:25.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233288+0000", 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233288+0000", 2024-09-16T04:42:25.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233289+0000", 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4169999999999999e-06 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233296+0000", 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3830000000000003e-06 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233318+0000", 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.145e-05 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233334+0000", 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6429000000000001e-05 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.342490+0000", 2024-09-16T04:42:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10915607099999999 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.342926+0000", 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043572400000000002 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.344296+0000", 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001369566 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:25.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.645708+0000", 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30141223700000003 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747663+0000", 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10195557800000001 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747820+0000", 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156539 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.983 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:5 3.5s0 3:a2d95112:::smithi16530038-40:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.237468+0000", 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.684598962999999, 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71584401900000005, 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.985 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:25.985 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237468+0000", 2024-09-16T04:42:25.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237468+0000", 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237471+0000", 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9440000000000001e-06 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237487+0000", 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5722999999999998e-05 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237532+0000", 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5116e-05 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237554+0000", 2024-09-16T04:42:25.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1766000000000001e-05 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.850151+0000", 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61259759800000002 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.850307+0000", 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015556599999999999 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.953312+0000", 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103005306 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=33, last_complete=65'23, committed=1, applied=1))", 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.240426+0000", 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.681640666, 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30497217100000001, 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.989 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240426+0000", 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240426+0000", 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240428+0000", 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5379999999999999e-06 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240437+0000", 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.149e-06 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240461+0000", 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3912000000000001e-05 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240476+0000", 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5543e-05 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545363+0000", 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30488681299999998 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545398+0000", 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5216000000000003e-05 2024-09-16T04:42:25.992 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=33, last_complete=65'23, committed=1, applied=1))", 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.242209+0000", 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.679858245, 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30325555999999998, 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.242209+0000", 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.242209+0000", 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.242210+0000", 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.61e-06 2024-09-16T04:42:25.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.242218+0000", 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0460000000000006e-06 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.242239+0000", 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0203999999999998e-05 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.242252+0000", 2024-09-16T04:42:25.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3720000000000001e-05 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545438+0000", 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30318529999999999 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545464+0000", 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6679999999999999e-05 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:25.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=33, last_complete=65'23, committed=1, applied=1))", 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.243164+0000", 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.678902612, 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40294550099999998, 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243164+0000", 2024-09-16T04:42:25.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243164+0000", 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243166+0000", 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.84e-06 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243176+0000", 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0152999999999999e-05 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243277+0000", 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101006 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243293+0000", 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5896000000000002e-05 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:25.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.645804+0000", 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402510545 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646110+0000", 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000306061 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.000 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=34, last_complete=65'24, committed=1, applied=1))", 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.243343+0000", 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.678723937000001, 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40290150200000002, 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243343+0000", 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243343+0000", 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243344+0000", 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.345e-06 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243353+0000", 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:42:26.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243376+0000", 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3419000000000001e-05 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.343476+0000", 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100099434 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646210+0000", 2024-09-16T04:42:26.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30273381399999999 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646244+0000", 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4866999999999997e-05 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=34, last_complete=65'24, committed=1, applied=1))", 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.247549+0000", 2024-09-16T04:42:26.004 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.674517792, 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49961050899999998, 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247549+0000", 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247549+0000", 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247551+0000", 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.95e-06 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247563+0000", 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2300000000000001e-05 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247590+0000", 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7016999999999999e-05 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247611+0000", 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0696e-05 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746819+0000", 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49920759799999997 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747160+0000", 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034094799999999999 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=34, last_complete=65'24, committed=1, applied=1))", 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.250731+0000", 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.671335885, 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49578053700000002, 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.008 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250731+0000", 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250731+0000", 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250733+0000", 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7689999999999999e-06 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250741+0000", 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5939999999999994e-06 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250766+0000", 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4206000000000002e-05 2024-09-16T04:42:26.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250780+0000", 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3963e-05 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746474+0000", 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49569433200000002 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746512+0000", 2024-09-16T04:42:26.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7673000000000003e-05 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.346174+0000", 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.575893279000001, 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40273614899999999, 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346174+0000", 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346174+0000", 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346175+0000", 2024-09-16T04:42:26.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.517e-06 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346184+0000", 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346208+0000", 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4063e-05 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346228+0000", 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9148999999999999e-05 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747539+0000", 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40131134099999999 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.748910+0000", 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001370806 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.346887+0000", 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.575179898, 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40035135100000002, 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346887+0000", 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346887+0000", 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346889+0000", 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8700000000000001e-06 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346898+0000", 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4360000000000008e-06 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346928+0000", 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9904e-05 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346950+0000", 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1460000000000001e-05 2024-09-16T04:42:26.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747213+0000", 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40026329399999999 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747238+0000", 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5386999999999999e-05 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.347600+0000", 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.574467096999999, 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39977881599999998, 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347600+0000", 2024-09-16T04:42:26.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347600+0000", 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347602+0000", 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.86e-06 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347612+0000", 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0530000000000001e-05 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347643+0000", 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1066999999999999e-05 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347664+0000", 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.048e-05 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747346+0000", 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39968240500000002 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747379+0000", 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2474000000000001e-05 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.434934+0000", 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.487132663000001, 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31236023899999998, 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434934+0000", 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434934+0000", 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434936+0000", 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-06 2024-09-16T04:42:26.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434946+0000", 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0723999999999999e-05 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434978+0000", 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1933000000000001e-05 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434999+0000", 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0109999999999999e-05 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747273+0000", 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31227403300000001 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747295+0000", 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1946e-05 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.444252+0000", 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.477815210999999, 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30324630899999999, 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:26.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.444252+0000", 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.444252+0000", 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.444252+0000", 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7700000000000004e-07 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.444256+0000", 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.856e-06 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.444265+0000", 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9590000000000001e-06 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.444272+0000", 2024-09-16T04:42:26.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0380000000000002e-06 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747475+0000", 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30320227199999999 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747498+0000", 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3507e-05 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:26.032 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:26.032 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:26.033 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:26.034 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:26.027+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:26.135 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:26.131+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:26.138 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T04:42:26.143 INFO:tasks.rados.rados.0.smithi165.stdout:249: done (4 left) 2024-09-16T04:42:26.143 INFO:tasks.rados.rados.0.smithi165.stdout:250: done (3 left) 2024-09-16T04:42:26.143 INFO:tasks.rados.rados.0.smithi165.stdout:255: done (2 left) 2024-09-16T04:42:26.143 INFO:tasks.rados.rados.0.smithi165.stdout:256: done (1 left) 2024-09-16T04:42:26.144 INFO:tasks.rados.rados.0.smithi165.stdout:257: done (0 left) 2024-09-16T04:42:26.144 INFO:tasks.rados.rados.0.smithi165.stdout:258: read oid 47 snap -1 2024-09-16T04:42:26.144 INFO:tasks.rados.rados.0.smithi165.stdout:258: expect (ObjNum 61 snap 3 seq_num 61) 2024-09-16T04:42:26.144 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 45 current snap is 20 2024-09-16T04:42:26.144 INFO:tasks.rados.rados.0.smithi165.stdout:259: seq_num 106 ranges {131072=622592,753664=442368,1196032=524288,1720320=196608} 2024-09-16T04:42:26.157 INFO:tasks.rados.rados.0.smithi165.stdout:259: writing smithi16530038-45 from 131072 to 753664 tid 1 2024-09-16T04:42:26.164 INFO:tasks.rados.rados.0.smithi165.stdout:259: writing smithi16530038-45 from 753664 to 1196032 tid 2 2024-09-16T04:42:26.165 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T04:42:26.173 INFO:tasks.rados.rados.0.smithi165.stdout:259: writing smithi16530038-45 from 1196032 to 1720320 tid 3 2024-09-16T04:42:26.173 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:26.174 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:259: writing smithi16530038-45 from 1720320 to 1916928 tid 4 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:260: read oid 49 snap 11 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:260: expect (ObjNum 59 snap 3 seq_num 59) 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:261: read oid 28 snap -1 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:261: expect (ObjNum 53 snap 1 seq_num 53) 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:262: read oid 11 snap -1 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:262: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:263: read oid 5 snap 16 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:263: expect deleted 2024-09-16T04:42:26.177 INFO:tasks.rados.rados.0.smithi165.stdout:264: setattr oid 48 current snap is 20 2024-09-16T04:42:26.182 INFO:tasks.rados.rados.0.smithi165.stdout:265: rollback oid 14 current snap is 20 2024-09-16T04:42:26.182 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 17 2024-09-16T04:42:26.182 INFO:tasks.rados.rados.0.smithi165.stdout:266: delete oid 18 current snap is 20 2024-09-16T04:42:26.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:26.235 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:26.231+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:26.336 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:26.331+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:26.364 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:26.364 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:26.364 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:26.364 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:26.364 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:26.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:42:26.378 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:26.437 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:26.435+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:26.486 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 136 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:42:26.488 INFO:tasks.rados.rados.0.smithi165.stdout:265: finishing rollback tid 0 to smithi16530038-14 2024-09-16T04:42:26.491 INFO:tasks.rados.rados.0.smithi165.stdout:260: expect (ObjNum 59 snap 3 seq_num 59) 2024-09-16T04:42:26.512 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:26.538 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:26.535+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:26.555 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:26.555 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s4 36/32 ECSubWrite(tid=8, reqid=client.4728.0:109, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 41.179489769, 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201785732, 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374507+0000", 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374649+0000", 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014185599999999999 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374672+0000", 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3277000000000001e-05 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374685+0000", 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2797e-05 2024-09-16T04:42:26.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574940+0000", 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20025501600000001 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574956+0000", 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5936000000000001e-05 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576258+0000", 2024-09-16T04:42:26.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001302388 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576291+0000", 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 36/32 ECSubWrite(tid=4, reqid=client.4728.0:112, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:26.560 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.980383441999997, 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20173359499999999, 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:26.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573614+0000", 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.114e-06 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574099+0000", 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048458300000000002 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574141+0000", 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2551e-05 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574168+0000", 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6899000000000001e-05 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674350+0000", 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100181464 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674362+0000", 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2138e-05 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775305+0000", 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100943171 2024-09-16T04:42:26.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775345+0000", 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0674999999999999e-05 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:164 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.656461008000001, 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40676020400000001, 2024-09-16T04:42:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 164 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897538+0000", 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0500000000000002e-06 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897554+0000", 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5886e-05 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897591+0000", 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6899e-05 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897617+0000", 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5922e-05 2024-09-16T04:42:26.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897727+0000", 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011011200000000001 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897835+0000", 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107889 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304263+0000", 2024-09-16T04:42:26.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40642834999999999 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304295+0000", 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1096000000000002e-05 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:165 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:26.570 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.656321320000004, 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20425574899999999, 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 165 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:26.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897676+0000", 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897685+0000", 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.183e-06 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897709+0000", 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4216e-05 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997840+0000", 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131417 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098015+0000", 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100174894 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098124+0000", 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108714 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101885+0000", 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037611459999999999 2024-09-16T04:42:26.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101930+0000", 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4675999999999997e-05 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:166 3.6s0 3.a17ee16 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.556105057000003, 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.306539124, 2024-09-16T04:42:26.575 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 166 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997892+0000", 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997905+0000", 2024-09-16T04:42:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2583e-05 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997926+0000", 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.101e-05 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098022+0000", 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100096478 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304408+0000", 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20638583099999999 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304429+0000", 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1132e-05 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.16s0 42/41 ECSubRead(tid=6, to_read={3:6877e850:::smithi16530038-5:head=0,20480,0}, subchunks={3:6877e850:::smithi16530038-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.245771990999998, 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20348143299999999, 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308260+0000", 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6318e-05 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308312+0000", 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2611999999999997e-05 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511485+0000", 2024-09-16T04:42:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20317243800000001 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511705+0000", 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220065 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.245096158000003, 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20303517200000001, 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308901+0000", 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308970+0000", 2024-09-16T04:42:26.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9240999999999997e-05 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309005+0000", 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4314999999999999e-05 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309023+0000", 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8879999999999999e-05 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511826+0000", 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202802133 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511934+0000", 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108756 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.244861366000002, 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20302847199999999, 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309135+0000", 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309242+0000", 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106959 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309260+0000", 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309270+0000", 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0339e-05 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512036+0000", 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20276591199999999 2024-09-16T04:42:26.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512162+0000", 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126675 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.14s4 43/41 ECSubWrite(tid=30, reqid=client.4728.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.214718963999999, 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20195774699999999, 2024-09-16T04:42:26.591 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339277+0000", 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.61e-07 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339289+0000", 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1533e-05 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339303+0000", 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4002000000000001e-05 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439391+0000", 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10008851100000001 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439604+0000", 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301100000000001 2024-09-16T04:42:26.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439617+0000", 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3076e-05 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541138+0000", 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10152072299999999 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541234+0000", 2024-09-16T04:42:26.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6130000000000003e-05 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 53/32 ECSubWrite(tid=89, reqid=client.4728.0:426, at_version=53'29, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.619800947000002, 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20199651099999999, 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934195+0000", 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:42:26.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934336+0000", 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014033399999999999 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934357+0000", 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1769000000000001e-05 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034519+0000", 2024-09-16T04:42:26.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016164599999999 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034685+0000", 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000166312 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034712+0000", 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6959999999999999e-05 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136138+0000", 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101425772 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136191+0000", 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2868000000000003e-05 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=115, reqid=client.4728.0:457, at_version=58'26, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:26.600 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.254639787999999, 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.240164188, 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:26.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:26.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5699999999999999e-07 2024-09-16T04:42:26.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299490+0000", 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013428299999999999 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299510+0000", 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437999999999999e-05 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299523+0000", 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3132000000000001e-05 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436652+0000", 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13712892300000001 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436659+0000", 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4410000000000004e-06 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539460+0000", 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102800512 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539520+0000", 2024-09-16T04:42:26.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0201999999999997e-05 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=122, reqid=client.4728.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=58'29))", 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.741513977, 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20212734700000001, 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812484+0000", 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3700000000000002e-06 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812511+0000", 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.775e-05 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812550+0000", 2024-09-16T04:42:26.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812567+0000", 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7186000000000001e-05 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812770+0000", 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202862 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812799+0000", 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8716000000000001e-05 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014571+0000", 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201771586 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014609+0000", 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8124999999999998e-05 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.301202208999999, 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.296481516, 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252794+0000", 2024-09-16T04:42:26.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253432+0000", 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063874400000000001 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253463+0000", 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0052999999999998e-05 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253487+0000", 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4335999999999999e-05 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344896+0000", 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.091408780999999995 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344925+0000", 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9490000000000001e-05 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549182+0000", 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.204257051 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549275+0000", 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2464000000000001e-05 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:26.615 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.295411353, 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29077158600000003, 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258586+0000", 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258848+0000", 2024-09-16T04:42:26.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026226500000000002 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258888+0000", 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0395999999999997e-05 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258917+0000", 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8371999999999999e-05 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.346992+0000", 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.088075353999999995 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.347031+0000", 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549288+0000", 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20225760800000001 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549356+0000", 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7222000000000001e-05 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.292544618999999, 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.288032389, 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.620 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261453+0000", 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.79e-06 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261566+0000", 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000113801 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261597+0000", 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0772999999999998e-05 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261613+0000", 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348618+0000", 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.087004695000000007 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348634+0000", 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6246e-05 2024-09-16T04:42:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549401+0000", 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200766994 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549483+0000", 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2150999999999993e-05 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.290849599, 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28676506400000001, 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.625 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:26.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263148+0000", 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.81e-06 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263180+0000", 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2019e-05 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263210+0000", 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0429999999999998e-05 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263226+0000", 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6093e-05 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548818+0000", 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28559150300000002 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548834+0000", 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6804e-05 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549811+0000", 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00097657100000000008 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549911+0000", 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9834000000000007e-05 2024-09-16T04:42:26.630 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds4 70/67 ECSubWrite(tid=146, reqid=osd.7.0:146, at_version=70'118, trim_to=0'0, roll_forward_to=70'116))", 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:20.558888+0000", 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.9951069209999996, 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20169158400000001, 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.631 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558888+0000", 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558888+0000", 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558890+0000", 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1519999999999999e-06 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558927+0000", 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7248e-05 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558940+0000", 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3352e-05 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.659049+0000", 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100108354 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.759316+0000", 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10026745099999999 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.759349+0000", 2024-09-16T04:42:26.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2923000000000003e-05 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.760523+0000", 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011739350000000001 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.760580+0000", 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7169000000000001e-05 2024-09-16T04:42:26.635 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.636 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/67 ECSubWrite(tid=159, reqid=client.4728.0:622, at_version=74'134, trim_to=0'0, roll_forward_to=72'132))", 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 2.080398872, 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.205798497, 2024-09-16T04:42:26.637 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:26.638 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473597+0000", 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3999999999999998e-07 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473620+0000", 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3383999999999998e-05 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473630+0000", 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473639+0000", 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9799999999999993e-06 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676469+0000", 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202829168 2024-09-16T04:42:26.640 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676503+0000", 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4239999999999997e-05 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679328+0000", 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028252860000000002 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679395+0000", 2024-09-16T04:42:26.641 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6773000000000006e-05 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.21s5,QREQUEST,e:74,from:2(0),reservation_nonce:2)", 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.617429934, 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30070015300000003, 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936567+0000", 2024-09-16T04:42:26.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.86e-06 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936576+0000", 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0559999999999994e-06 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936605+0000", 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8632999999999998e-05 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.036785+0000", 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10017992000000001 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237143+0000", 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20035802999999999 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237193+0000", 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0121999999999999e-05 2024-09-16T04:42:26.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237266+0000", 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2532000000000003e-05 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.bs1 76/67 ECSubRead(tid=164, to_read={3:d388eed4:::smithi16530038-28:head=0,339968,0}, subchunks={3:d388eed4:::smithi16530038-28:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.276088793, 2024-09-16T04:42:26.646 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22646649499999999, 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:26.647 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277908+0000", 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7439999999999999e-06 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277925+0000", 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7084000000000001e-05 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:26.648 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277960+0000", 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4208999999999997e-05 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277978+0000", 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8219e-05 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504200+0000", 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22622223899999999 2024-09-16T04:42:26.649 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504373+0000", 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000173 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:26.650 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:26.651 INFO:tasks.rados.rados.0.smithi165.stdout:258: expect (ObjNum 61 snap 3 seq_num 61) 2024-09-16T04:42:26.652 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:26.653 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:26.653 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:26.647+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:26.671 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:26.684 INFO:tasks.rados.rados.0.smithi165.stdout:259: finishing write tid 1 to smithi16530038-45 2024-09-16T04:42:26.685 INFO:tasks.rados.rados.0.smithi165.stdout:259: finishing write tid 2 to smithi16530038-45 2024-09-16T04:42:26.690 INFO:tasks.rados.rados.0.smithi165.stdout:265: finishing rollback tid 1 to smithi16530038-14 2024-09-16T04:42:26.691 INFO:tasks.rados.rados.0.smithi165.stdout:265: finishing rollback tid 2 to smithi16530038-14 2024-09-16T04:42:26.691 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 37 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T04:42:26.691 INFO:tasks.rados.rados.0.smithi165.stdout:261: expect (ObjNum 53 snap 1 seq_num 53) 2024-09-16T04:42:26.695 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:26.731 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:26.731 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:26.749 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:26.750 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:26.752 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:26.747+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:26.802 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:26.802 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:26.815 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:26.815 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:26.816 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:26.838 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:26.866 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:26.866 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:26.863+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:26.883 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:26.884 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:26.886 INFO:tasks.rados.rados.0.smithi165.stdout:259: finishing write tid 3 to smithi16530038-45 2024-09-16T04:42:26.920 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:26.920 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:26.954 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:26.951+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:26.969 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:26.969 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:26.984 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:27.009 INFO:tasks.rados.rados.0.smithi165.stdout:262: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-16T04:42:27.031 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:27.032 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:27.055 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:27.051+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:27.057 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:27.058 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:27.076 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:27.077 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:27.096 INFO:tasks.rados.rados.0.smithi165.stdout:258: done (8 left) 2024-09-16T04:42:27.096 INFO:tasks.rados.rados.0.smithi165.stdout:260: done (7 left) 2024-09-16T04:42:27.096 INFO:tasks.rados.rados.0.smithi165.stdout:261: done (6 left) 2024-09-16T04:42:27.096 INFO:tasks.rados.rados.0.smithi165.stdout:262: done (5 left) 2024-09-16T04:42:27.097 INFO:tasks.rados.rados.0.smithi165.stdout:263: done (4 left) 2024-09-16T04:42:27.097 INFO:tasks.rados.rados.0.smithi165.stdout:264: done (3 left) 2024-09-16T04:42:27.097 INFO:tasks.rados.rados.0.smithi165.stdout:265: done (2 left) 2024-09-16T04:42:27.097 INFO:tasks.rados.rados.0.smithi165.stdout:266: done (1 left) 2024-09-16T04:42:27.097 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 34 current snap is 20 2024-09-16T04:42:27.097 INFO:tasks.rados.rados.0.smithi165.stdout:267: seq_num 108 ranges {0=655360,655360=606208,1261568=475136,1736704=81920} 2024-09-16T04:42:27.097 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:27.098 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:27.107 INFO:tasks.rados.rados.0.smithi165.stdout:267: writing smithi16530038-34 from 0 to 655360 tid 1 2024-09-16T04:42:27.118 INFO:tasks.rados.rados.0.smithi165.stdout:267: writing smithi16530038-34 from 655360 to 1261568 tid 2 2024-09-16T04:42:27.119 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:27.126 INFO:tasks.rados.rados.0.smithi165.stdout:267: writing smithi16530038-34 from 1261568 to 1736704 tid 3 2024-09-16T04:42:27.127 INFO:tasks.rados.rados.0.smithi165.stdout:267: writing smithi16530038-34 from 1736704 to 1818624 tid 4 2024-09-16T04:42:27.127 INFO:tasks.rados.rados.0.smithi165.stdout:268: setattr oid 41 current snap is 20 2024-09-16T04:42:27.132 INFO:tasks.rados.rados.0.smithi165.stdout:269: setattr oid 31 current snap is 20 2024-09-16T04:42:27.137 INFO:tasks.rados.rados.0.smithi165.stdout:270: snap_remove snap 18 2024-09-16T04:42:27.140 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:27.140 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:27.154 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:27.154 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:27.156 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:27.151+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:27.170 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:27.256 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:27.251+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:27.348 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:27.348 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:27.358 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:27.355+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:27.360 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T04:42:27.361 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:42:27.363 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:42:27.365 INFO:tasks.rados.rados.0.smithi165.stdout:270: done (4 left) 2024-09-16T04:42:27.365 INFO:tasks.rados.rados.0.smithi165.stdout:271: read oid 8 snap -1 2024-09-16T04:42:27.365 INFO:tasks.rados.rados.0.smithi165.stdout:271: expect (ObjNum 74 snap 5 seq_num 74) 2024-09-16T04:42:27.365 INFO:tasks.rados.rados.0.smithi165.stdout:272: read oid 15 snap -1 2024-09-16T04:42:27.365 INFO:tasks.rados.rados.0.smithi165.stdout:272: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:27.365 INFO:tasks.rados.rados.0.smithi165.stdout:273: read oid 2 snap -1 2024-09-16T04:42:27.365 INFO:tasks.rados.rados.0.smithi165.stdout:273: expect deleted 2024-09-16T04:42:27.366 INFO:tasks.rados.rados.0.smithi165.stdout:274: snap_remove snap 19 2024-09-16T04:42:27.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T04:42:27.459 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:27.455+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:27.488 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:27.488 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:27.506 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:42:27.560 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:27.555+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:27.629 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:27.629 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:27.644 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:42:27.661 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:27.659+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:27.762 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:27.759+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:27.775 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:27.775 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:27.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:42:27.863 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:27.859+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:27.914 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:27.914 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:27.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:42:27.966 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:27.963+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:28.051 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:28.051 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:28.065 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:42:28.067 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:28.063+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:28.168 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:28.163+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:28.199 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:28.200 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:28.215 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:42:28.269 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:28.263+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:28.336 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:28.336 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:28.350 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:42:28.364 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T04:42:28.370 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:28.367+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:28.372 INFO:tasks.rados.rados.0.smithi165.stdout:274: done (7 left) 2024-09-16T04:42:28.372 INFO:tasks.rados.rados.0.smithi165.stdout:275: copy_from oid 50 from oid 47 current snap is 20 2024-09-16T04:42:28.372 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing write tid 1 to smithi16530038-34 2024-09-16T04:42:28.372 INFO:tasks.rados.rados.0.smithi165.stdout:259: finishing write tid 4 to smithi16530038-45 2024-09-16T04:42:28.372 INFO:tasks.rados.rados.0.smithi165.stdout:259: finishing write tid 5 to smithi16530038-45 2024-09-16T04:42:28.372 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 43 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: finishing write tid 6 to smithi16530038-45 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: oid 45 updating version 0 to 38 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: oid 45 updating version 38 to 39 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: oid 45 version 39 is already newer than 35 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: oid 45 version 39 is already newer than 37 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: oid 45 version 39 is already newer than 36 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 39 (ObjNum 106 snap 20 seq_num 106) dirty exists 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: left oid 45 (ObjNum 106 snap 20 seq_num 106) 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 37 (ObjNum 83 snap 12 seq_num 83) dirty exists 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing write tid 2 to smithi16530038-34 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:259: done (7 left) 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:268: done (6 left) 2024-09-16T04:42:28.373 INFO:tasks.rados.rados.0.smithi165.stdout:269: done (5 left) 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:273: done (4 left) 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:276: read oid 48 snap -1 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:276: expect (ObjNum 54 snap 1 seq_num 54) 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:277: read oid 16 snap -1 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:277: expect (ObjNum 75 snap 6 seq_num 75) 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:278: read oid 13 snap 15 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:278: expect (ObjNum 86 snap 14 seq_num 86) 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 40 current snap is 20 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing write tid 3 to smithi16530038-34 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing write tid 4 to smithi16530038-34 2024-09-16T04:42:28.374 INFO:tasks.rados.rados.0.smithi165.stdout:272: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:28.378 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing write tid 5 to smithi16530038-34 2024-09-16T04:42:28.378 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing write tid 6 to smithi16530038-34 2024-09-16T04:42:28.378 INFO:tasks.rados.rados.0.smithi165.stdout:267: oid 34 updating version 0 to 61 2024-09-16T04:42:28.378 INFO:tasks.rados.rados.0.smithi165.stdout:267: oid 34 version 61 is already newer than 59 2024-09-16T04:42:28.378 INFO:tasks.rados.rados.0.smithi165.stdout:267: oid 34 version 61 is already newer than 60 2024-09-16T04:42:28.378 INFO:tasks.rados.rados.0.smithi165.stdout:267: oid 34 updating version 61 to 63 2024-09-16T04:42:28.378 INFO:tasks.rados.rados.0.smithi165.stdout:267: oid 34 version 63 is already newer than 62 2024-09-16T04:42:28.379 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 63 (ObjNum 108 snap 20 seq_num 108) dirty exists 2024-09-16T04:42:28.379 INFO:tasks.rados.rados.0.smithi165.stdout:267: left oid 34 (ObjNum 108 snap 20 seq_num 108) 2024-09-16T04:42:28.379 INFO:tasks.rados.rados.0.smithi165.stdout:271: expect (ObjNum 74 snap 5 seq_num 74) 2024-09-16T04:42:28.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T04:42:28.463 INFO:tasks.rados.rados.0.smithi165.stdout:279: seq_num 109 ranges {671744=671744,1343488=753664,2097152=409600} 2024-09-16T04:42:28.471 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:28.467+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:28.482 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:28.482 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:28.487 INFO:tasks.rados.rados.0.smithi165.stdout:279: writing smithi16530038-40 from 671744 to 1343488 tid 1 2024-09-16T04:42:28.498 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:42:28.499 INFO:tasks.rados.rados.0.smithi165.stdout:279: writing smithi16530038-40 from 1343488 to 2097152 tid 2 2024-09-16T04:42:28.507 INFO:tasks.rados.rados.0.smithi165.stdout:279: writing smithi16530038-40 from 2097152 to 2506752 tid 3 2024-09-16T04:42:28.507 INFO:tasks.rados.rados.0.smithi165.stdout:267: done (7 left) 2024-09-16T04:42:28.507 INFO:tasks.rados.rados.0.smithi165.stdout:271: done (6 left) 2024-09-16T04:42:28.508 INFO:tasks.rados.rados.0.smithi165.stdout:272: done (5 left) 2024-09-16T04:42:28.508 INFO:tasks.rados.rados.0.smithi165.stdout:280: snap_remove snap 17 2024-09-16T04:42:28.574 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:28.567+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:28.665 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:28.666 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:28.674 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:28.671+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:28.680 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:42:28.774 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:28.771+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:28.839 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:28.839 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:28.854 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:42:28.884 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:28.871+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:28.980 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:28.975+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:28.987 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:28.987 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:29.001 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:42:29.080 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:29.075+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:29.160 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:29.175 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:29.175 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:29.179 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:29.175+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:29.190 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:42:29.280 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:29.275+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:29.310 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:29.310 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:29.319 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:29.323 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:42:29.340 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:29.361 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:29.380 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T04:42:29.381 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:29.375+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:29.383 INFO:tasks.rados.rados.0.smithi165.stdout:280: done (5 left) 2024-09-16T04:42:29.383 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 35 current snap is 20 2024-09-16T04:42:29.384 INFO:tasks.rados.rados.0.smithi165.stdout:281: seq_num 110 ranges {573440=688128,1261568=507904,1769472=655360} 2024-09-16T04:42:29.387 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:29.387 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:29.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:42:29.405 INFO:tasks.rados.rados.0.smithi165.stdout:281: writing smithi16530038-35 from 573440 to 1261568 tid 1 2024-09-16T04:42:29.408 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T04:42:29.410 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:29.411 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:29.413 INFO:tasks.rados.rados.0.smithi165.stdout:281: writing smithi16530038-35 from 1261568 to 1769472 tid 2 2024-09-16T04:42:29.423 INFO:tasks.rados.rados.0.smithi165.stdout:281: writing smithi16530038-35 from 1769472 to 2424832 tid 3 2024-09-16T04:42:29.423 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 23 current snap is 20 2024-09-16T04:42:29.424 INFO:tasks.rados.rados.0.smithi165.stdout:282: seq_num 111 ranges {0=704512,704512=655360,1359872=507904} 2024-09-16T04:42:29.434 INFO:tasks.rados.rados.0.smithi165.stdout:282: writing smithi16530038-23 from 0 to 704512 tid 1 2024-09-16T04:42:29.444 INFO:tasks.rados.rados.0.smithi165.stdout:282: writing smithi16530038-23 from 704512 to 1359872 tid 2 2024-09-16T04:42:29.444 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:29.445 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:29.451 INFO:tasks.rados.rados.0.smithi165.stdout:282: writing smithi16530038-23 from 1359872 to 1867776 tid 3 2024-09-16T04:42:29.451 INFO:tasks.rados.rados.0.smithi165.stdout:283: rollback oid 18 current snap is 20 2024-09-16T04:42:29.451 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 18 to 15 2024-09-16T04:42:29.451 INFO:tasks.rados.rados.0.smithi165.stdout:278: expect (ObjNum 86 snap 14 seq_num 86) 2024-09-16T04:42:29.453 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:29.453 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:29.466 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:29.467 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:29.468 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:42:29.482 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:29.479+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:29.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:29.509 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:29.510 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:29.529 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:29.551 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:29.552 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:29.554 INFO:tasks.rados.rados.0.smithi165.stdout:277: expect (ObjNum 75 snap 6 seq_num 75) 2024-09-16T04:42:29.572 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:29.572 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:29.583 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:29.579+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:29.599 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:29.599 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:275: finishing copy_from racing read to smithi16530038-50 2024-09-16T04:42:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 1 to smithi16530038-40 2024-09-16T04:42:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 2 to smithi16530038-40 2024-09-16T04:42:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:276: expect (ObjNum 54 snap 1 seq_num 54) 2024-09-16T04:42:29.615 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:42:29.620 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:29.620 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:29.644 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:29.645 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:275: finishing copy_from to smithi16530038-50 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 20 (ObjNum 61 snap 3 seq_num 61) dirty exists 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 3 to smithi16530038-40 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 4 to smithi16530038-40 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 5 to smithi16530038-40 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:279: oid 40 updating version 0 to 44 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:279: oid 40 version 44 is already newer than 43 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:279: oid 40 updating version 44 to 46 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:279: oid 40 version 46 is already newer than 45 2024-09-16T04:42:29.661 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 46 (ObjNum 109 snap 20 seq_num 109) dirty exists 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:279: left oid 40 (ObjNum 109 snap 20 seq_num 109) 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:275: done (7 left) 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:276: done (6 left) 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:277: done (5 left) 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:278: done (4 left) 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:279: done (3 left) 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:284: rollback oid 3 current snap is 20 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 11 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:285: rmattr oid 20 current snap is 20 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:285: done (4 left) 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:286: rollback oid 13 current snap is 20 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 15 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:287: rmattr oid 47 current snap is 20 2024-09-16T04:42:29.662 INFO:tasks.rados.rados.0.smithi165.stdout:287: done (5 left) 2024-09-16T04:42:29.663 INFO:tasks.rados.rados.0.smithi165.stdout:288: copy_from oid 39 from oid 27 current snap is 20 2024-09-16T04:42:29.663 INFO:tasks.rados.rados.0.smithi165.stdout:289: rollback oid 9 current snap is 20 2024-09-16T04:42:29.663 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 15 2024-09-16T04:42:29.663 INFO:tasks.rados.rados.0.smithi165.stdout:290: snap_remove snap 16 2024-09-16T04:42:29.708 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:29.703+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:29.785 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:29.779+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:29.787 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:29.787 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:29.787 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:29.787 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:29.787 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:29.788 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:29.788 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:29.802 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:42:29.814 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:29.815 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:29.885 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:29.879+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:29.977 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 65.598653978000002, 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1755266710000001, 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:29.978 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:29.979 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374332+0000", 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8059999999999999e-06 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374345+0000", 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2529e-05 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374365+0000", 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0695000000000001e-05 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374383+0000", 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7569000000000001e-05 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374514+0000", 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013070099999999999 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374544+0000", 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0731000000000001e-05 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.343896+0000", 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.969351224 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344169+0000", 2024-09-16T04:42:29.982 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027390100000000001 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344583+0000", 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041308600000000001 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.346047+0000", 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014642699999999999 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:29.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549695+0000", 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203648463 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549791+0000", 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6137999999999993e-05 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549856+0000", 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:29.984 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 57.352237799000001, 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51346886800000002, 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 111 2024-09-16T04:42:29.985 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:29.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620747+0000", 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620757+0000", 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0316000000000001e-05 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:29.987 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620778+0000", 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620792+0000", 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4362000000000001e-05 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928510+0000", 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.307717717 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.988 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928629+0000", 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011919 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928822+0000", 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019246199999999999 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930635+0000", 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00181292 2024-09-16T04:42:29.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134065+0000", 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20343018399999999 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134149+0000", 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4635000000000005e-05 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134214+0000", 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4973000000000003e-05 2024-09-16T04:42:29.990 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 57.352167436000002, 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51366009000000001, 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:29.991 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 112 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.992 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620817+0000", 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621035+0000", 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021731100000000001 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621061+0000", 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6744999999999999e-05 2024-09-16T04:42:29.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621082+0000", 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1061999999999999e-05 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929460+0000", 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.308377346 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929595+0000", 2024-09-16T04:42:29.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013512600000000001 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929798+0000", 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020277199999999999 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932465+0000", 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002667862 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134319+0000", 2024-09-16T04:42:29.995 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20185381599999999 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134407+0000", 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7894999999999998e-05 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134476+0000", 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8829000000000001e-05 2024-09-16T04:42:29.996 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 57.351867493, 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51357863199999998, 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:29.997 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 113 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:29.998 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621118+0000", 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.689e-06 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621129+0000", 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621164+0000", 2024-09-16T04:42:29.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5367999999999997e-05 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621182+0000", 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8352e-05 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930689+0000", 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.309507015 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930800+0000", 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000110871 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931026+0000", 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000226293 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.933569+0000", 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002542786 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.001 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134566+0000", 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200996272 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134640+0000", 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4412000000000005e-05 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134694+0000", 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4524000000000002e-05 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 57.351770746, 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51372108500000002, 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 114 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621214+0000", 2024-09-16T04:42:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621225+0000", 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1348999999999999e-05 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621245+0000", 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9966000000000002e-05 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621259+0000", 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4102e-05 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931642+0000", 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31038274300000002 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931776+0000", 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133552 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931973+0000", 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197262 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.033893+0000", 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101919751 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134774+0000", 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10088169600000001 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134847+0000", 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2309000000000002e-05 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134934+0000", 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6924999999999999e-05 2024-09-16T04:42:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 57.351698937000002, 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51387291300000004, 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 115 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621286+0000", 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.26e-06 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621492+0000", 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020643500000000001 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621517+0000", 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:42:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621534+0000", 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6515999999999999e-05 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932515+0000", 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31098146799999998 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932631+0000", 2024-09-16T04:42:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011636 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932814+0000", 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000182406 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134236+0000", 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201422618 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135029+0000", 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079253199999999998 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135102+0000", 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3286000000000003e-05 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135157+0000", 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5043000000000001e-05 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 57.351417118999997, 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.717172162, 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 116 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621568+0000", 2024-09-16T04:42:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621606+0000", 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8167999999999998e-05 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621628+0000", 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1742e-05 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621643+0000", 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5696e-05 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235313+0000", 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61366968300000002 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235449+0000", 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136036 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235657+0000", 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207405 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237049+0000", 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001392552 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338582+0000", 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101532626 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338670+0000", 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7888999999999998e-05 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338738+0000", 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8769000000000005e-05 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 57.351309129000001, 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.81671015999999996, 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 117 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.021 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621676+0000", 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4440000000000001e-06 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621686+0000", 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0845e-05 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621706+0000", 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9168999999999999e-05 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621723+0000", 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7538999999999999e-05 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236241+0000", 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61451815600000004 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236372+0000", 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013062300000000001 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236574+0000", 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020165600000000001 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237570+0000", 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099642999999999997 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438224+0000", 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20065369 2024-09-16T04:42:30.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438317+0000", 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3624999999999999e-05 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438384+0000", 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6983000000000006e-05 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 55.678235151999999, 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.508075677, 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 138 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294750+0000", 2024-09-16T04:42:30.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294760+0000", 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294779+0000", 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8941999999999999e-05 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.029 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294794+0000", 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5038999999999999e-05 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405306+0000", 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11051159300000001 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405426+0000", 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011977 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405611+0000", 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000185217 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709561+0000", 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394976699999998 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802549+0000", 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092988137999999998 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802707+0000", 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015861799999999999 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802824+0000", 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116517 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.032 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:30.033 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 55.678165028999999, 2024-09-16T04:42:30.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60756662299999997, 2024-09-16T04:42:30.033 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.034 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.034 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 139 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294820+0000", 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4330000000000001e-06 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294830+0000", 2024-09-16T04:42:30.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0023e-05 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294850+0000", 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0622e-05 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294864+0000", 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4042e-05 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406110+0000", 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11124526 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406222+0000", 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111871 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406398+0000", 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017598000000000001 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709577+0000", 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30317971500000002 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902101+0000", 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19252423499999999 2024-09-16T04:42:30.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902267+0000", 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016537499999999999 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902385+0000", 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011806700000000001 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 55.678072901, 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60781196900000001, 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 140 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.041 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294912+0000", 2024-09-16T04:42:30.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.407e-06 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295040+0000", 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012836600000000001 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295062+0000", 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1682e-05 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295083+0000", 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1109000000000001e-05 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507008+0000", 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21192486699999999 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507149+0000", 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141042 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507369+0000", 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022046500000000001 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710298+0000", 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202928308 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902525+0000", 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.192226958 2024-09-16T04:42:30.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902611+0000", 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6274999999999996e-05 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902722+0000", 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011149 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 55.677845320000003, 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70811914799999998, 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 141 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295140+0000", 2024-09-16T04:42:30.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295152+0000", 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2053e-05 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295172+0000", 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0282e-05 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295191+0000", 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8759000000000001e-05 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508486+0000", 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.213295545 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508630+0000", 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014419899999999999 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508824+0000", 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019358600000000001 2024-09-16T04:42:30.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710318+0000", 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20149381 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.002981+0000", 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.29266302500000002 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003144+0000", 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016330500000000001 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003257+0000", 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113064 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.055 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.013388591999998, 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51415646199999998, 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 159 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959597+0000", 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9190000000000002e-06 2024-09-16T04:42:30.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959609+0000", 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959629+0000", 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0035e-05 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.058 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959645+0000", 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5786000000000001e-05 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167529+0000", 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.207883768 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167762+0000", 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023295499999999999 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.168134+0000", 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037181099999999999 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.270830+0000", 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102696683 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473585+0000", 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20275470700000001 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473677+0000", 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1772000000000002e-05 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473751+0000", 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4228999999999997e-05 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.061 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.013307830000002, 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51420228400000001, 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.062 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 160 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959677+0000", 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959687+0000", 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959711+0000", 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3811999999999998e-05 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959727+0000", 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169193+0000", 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209466241 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169340+0000", 2024-09-16T04:42:30.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000147279 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169545+0000", 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000205167 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374117+0000", 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20457175999999999 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.066 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473763+0000", 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099646306000000004 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473828+0000", 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4046000000000003e-05 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473878+0000", 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0241000000000002e-05 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.013230550999999, 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51424688600000001, 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.068 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 161 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959754+0000", 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959920+0000", 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166088 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959944+0000", 2024-09-16T04:42:30.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959963+0000", 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9176000000000001e-05 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371049+0000", 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41108644500000002 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.071 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371185+0000", 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013619899999999999 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371395+0000", 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020912 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.377708+0000", 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006313173 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473891+0000", 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.096182904 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473949+0000", 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8016999999999997e-05 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474000+0000", 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:42:30.073 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.012914793999997, 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51552797100000003, 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.074 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 162 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960070+0000", 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960077+0000", 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6830000000000008e-06 2024-09-16T04:42:30.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960091+0000", 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3653e-05 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960101+0000", 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7629999999999995e-06 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372554+0000", 2024-09-16T04:42:30.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41245356599999999 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372785+0000", 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023023100000000001 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.373156+0000", 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000371601 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474010+0000", 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100854291 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475328+0000", 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001317304 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475477+0000", 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014954899999999999 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475596+0000", 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011913699999999999 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.012866566, 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51788103100000005, 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.080 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 163 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960118+0000", 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960124+0000", 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960134+0000", 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0115999999999999e-05 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960144+0000", 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374237+0000", 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41409377600000002 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374462+0000", 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022478600000000001 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374797+0000", 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000334586 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474018+0000", 2024-09-16T04:42:30.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099220947000000004 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477813+0000", 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003795171 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477921+0000", 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010822100000000001 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477998+0000", 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6662999999999994e-05 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.012827352000002, 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51822496799999995, 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.086 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 164 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:30.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960157+0000", 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7399999999999991e-07 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960241+0000", 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3677999999999994e-05 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960256+0000", 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5206e-05 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960267+0000", 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1372999999999999e-05 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375650+0000", 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41538323100000002 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375885+0000", 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000234758 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.376293+0000", 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040777599999999999 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474024+0000", 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097731440000000003 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478113+0000", 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004088225 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478261+0000", 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014825 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478381+0000", 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012005700000000001 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.012701278999998, 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51862660500000002, 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 165 2024-09-16T04:42:30.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960283+0000", 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960289+0000", 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.49e-06 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960301+0000", 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2585999999999999e-05 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960309+0000", 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472635+0000", 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51232596699999999 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472771+0000", 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136243 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473033+0000", 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026155699999999999 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474539+0000", 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015065 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478656+0000", 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041164319999999997 2024-09-16T04:42:30.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478793+0000", 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013757299999999999 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478909+0000", 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011530400000000001 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.097 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:153 3.3s0 3:c31e3ec6:::smithi16530038-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 43.683402147000002, 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.511692229, 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 153 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289583+0000", 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.04e-06 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289590+0000", 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4899999999999997e-06 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289610+0000", 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9814999999999999e-05 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289625+0000", 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5553000000000002e-05 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.296844+0000", 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072185369999999997 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297045+0000", 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020125199999999999 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599740+0000", 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.302695037 2024-09-16T04:42:30.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599866+0000", 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012644500000000001 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.801273+0000", 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20140706 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:30.102 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:30.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:42:30.103 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:30.031+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:30.135 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:30.131+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:30.235 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:30.235 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:30.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.235 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.21s0 79/69 ECSubWriteReply(tid=113, last_complete=79'33, committed=1, applied=1))", 2024-09-16T04:42:30.235 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.167401+0000", 2024-09-16T04:42:30.235 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.066651555000000001, 2024-09-16T04:42:30.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.066691492000000005, 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.167401+0000", 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.167401+0000", 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.167401+0000", 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0299999999999998e-07 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.167417+0000", 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5553000000000002e-05 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.167442+0000", 2024-09-16T04:42:30.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5219999999999999e-05 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.167456+0000", 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3720000000000001e-05 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 1 2024-09-16T04:42:30.238 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:30.239 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:30.231+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:30.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:42:30.337 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:30.331+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:30.372 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:30.372 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:30.372 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:30.372 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:30.372 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:30.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:42:30.390 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:42:30.390 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:290: done (7 left) 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:291: snap_create 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:283: finishing rollback tid 1 to smithi16530038-18 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 95 (ObjNum 81 snap 0 seq_num 2) dirty dne 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:282: finishing write tid 1 to smithi16530038-23 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:281: finishing write tid 1 to smithi16530038-35 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:282: finishing write tid 2 to smithi16530038-23 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:281: finishing write tid 2 to smithi16530038-35 2024-09-16T04:42:30.397 INFO:tasks.rados.rados.0.smithi165.stdout:281: finishing write tid 3 to smithi16530038-35 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:281: finishing write tid 4 to smithi16530038-35 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:288: finishing copy_from racing read to smithi16530038-39 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:282: finishing write tid 3 to smithi16530038-23 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:282: finishing write tid 4 to smithi16530038-23 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:284: finishing rollback tid 0 to smithi16530038-3 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:284: finishing rollback tid 1 to smithi16530038-3 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:284: finishing rollback tid 2 to smithi16530038-3 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 67 (ObjNum 70 snap 5 seq_num 70) dirty exists 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:289: finishing rollback tid 0 to smithi16530038-9 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:289: finishing rollback tid 1 to smithi16530038-9 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:281: finishing write tid 5 to smithi16530038-35 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:281: oid 35 updating version 0 to 14 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:281: oid 35 updating version 14 to 15 2024-09-16T04:42:30.398 INFO:tasks.rados.rados.0.smithi165.stdout:281: oid 35 version 15 is already newer than 13 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:281: oid 35 version 15 is already newer than 12 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 15 (ObjNum 110 snap 20 seq_num 110) dirty exists 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:281: left oid 35 (ObjNum 110 snap 20 seq_num 110) 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:288: finishing copy_from to smithi16530038-39 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:289: finishing rollback tid 2 to smithi16530038-9 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 69 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:282: finishing write tid 5 to smithi16530038-23 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:282: oid 23 updating version 0 to 56 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:282: oid 23 version 56 is already newer than 53 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:282: oid 23 version 56 is already newer than 55 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:282: oid 23 version 56 is already newer than 54 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 56 (ObjNum 111 snap 20 seq_num 111) dirty exists 2024-09-16T04:42:30.399 INFO:tasks.rados.rados.0.smithi165.stdout:282: left oid 23 (ObjNum 111 snap 20 seq_num 111) 2024-09-16T04:42:30.400 INFO:tasks.rados.rados.0.smithi165.stdout:286: finishing rollback tid 0 to smithi16530038-13 2024-09-16T04:42:30.400 INFO:tasks.rados.rados.0.smithi165.stdout:286: finishing rollback tid 1 to smithi16530038-13 2024-09-16T04:42:30.418 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T04:42:30.438 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:30.435+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:30.538 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:30.535+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:30.560 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:30.560 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:30.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:30.560 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:30.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.560 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:108 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.192312921999999, 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60644887800000002, 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 108 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362769+0000", 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.364e-06 2024-09-16T04:42:30.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362776+0000", 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9789999999999996e-06 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362795+0000", 2024-09-16T04:42:30.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8569000000000001e-05 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362811+0000", 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6084999999999999e-05 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369499+0000", 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066885180000000001 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369745+0000", 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024616900000000002 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669288+0000", 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29954281100000002 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669425+0000", 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013708200000000001 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.969217+0000", 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29979130100000001 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:365 3.es0 3:741a05b8:::smithi16530038-36:head [create,setxattr YsO`TUrrXaTptX`eXscONOt_]dcP^Xa]S^TWWiqm (11788) in=40b,setxattr oi^cYciifcdaOrObmV`rpY]_[\\_XRNasd^fYcUiQ (1017) in=40b,setxattr NnlWipS`S[obrmTjfOrXNR_U^RaXm^RTUZSjeTPn (1683) in=40b,setxattr `Rdq]pmXkmok]RYTVc\\kl_fsshipieap^dmiSalQ (9120) in=40b,setxattr knfoakkVZXZZ]kbe_aTg^SsRaeWmatrnY_XYmNTW (19016) in=40b,setxattr scTN] (490) in=5b,setxattr PO]ZkWnlNifWZXRn_chZPacZ^ecXSeS]XbpaUedS (1366) in=40b,setxattr iblrYV^QO]RQb (490) in=13b,setxattr PhO_UPjl^dXjhPPlrpqkqVeccWOOR_a_`_]jrTio (1507) in=40b,setxattr UZTrkNZltdZkQ]toZYlWWldlYaqUeeaUptZhU`jb (876) in=40b,setxattr \\gmX]bkoXbYR[m\\f]gtseQ]SjNsrN]bQcaYflTOm (526) in=40b,setxattr UP_asas]_aYSd^sUrb[^eUm]neUY\\gtgcqWqipQ` (1226) in=40b,setxattr RbnfoadlmU (490) in=10b,setxattr jXdNonknr[NtSaQ\\pNdNYmVQcc]SoYrrom\\WnZPa (1122) in=40b,setxattr ]oZ_lqdXiNU`nUS\\^_nlTisrNtadtmge[^VY[rce (1086) in=40b,setxattr RcmOsXpXhRW\\o_cOt\\W_b[OZQahkUpfe]i\\k]rYs (1894) in=40b,setxattr YmQiO^S[Ok\\XN (490) in=13b,setxattr jeqjjZWTOTSkPsbNo\\]tYekhQcfX`_T\\_jpUQsTg (1295) in=40b,setxattr pYXdVeXTn`gtW`QWoQac^OYrt]kicggO[\\katbaZ (1507) in=40b,setxattr UWQrTe]`QdQdW`Wc_^t`Zg^^hVc[VUa`liQo[NXO (1015) in=40b] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:59.166696+0000", 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.388384153000001, 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40577269700000002, 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.566 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 365 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166696+0000", 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166696+0000", 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166699+0000", 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5330000000000001e-06 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166838+0000", 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013858800000000001 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.166899+0000", 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1329999999999997e-05 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.266993+0000", 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100094238 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.267155+0000", 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016149299999999999 2024-09-16T04:42:30.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.267480+0000", 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032509299999999999 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.270800+0000", 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003320192 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.275086+0000", 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0042863980000000003 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.572282+0000", 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29719557400000002 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:59.572469+0000", 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018725799999999999 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.571 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:507 3.es0 3:7777e846:::smithi16530038-17:head [copy-from ver 21] snapc 10=[10,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.283257983999999, 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.526543015, 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.572 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 507 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271824+0000", 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7990000000000001e-06 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271832+0000", 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271852+0000", 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9194999999999998e-05 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.371991+0000", 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10013957800000001 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372175+0000", 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000183325 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372198+0000", 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3306e-05 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472740+0000", 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100542016 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472902+0000", 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016220399999999999 2024-09-16T04:42:30.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692422+0000", 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21951984699999999 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.796322+0000", 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10389983699999999 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798260+0000", 2024-09-16T04:42:30.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001938103 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798366+0000", 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010582900000000001 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:529 3.es0 3:741a05b8:::smithi16530038-36:head [create,setxattr `k]abUjc_kPO]\\rQ[m\\eRa\\f]aW]pgQYXV]gVkgc (1435) in=40b,setxattr qen`h]Vg]rpmXc_]tPY^Xpfd (1823) in=24b,setxattr [idOh[aNZjbdcYS\\eZaVVetqi[ric_V (1823) in=31b,setxattr daSce\\mdeZpfaaqjVU^^]PU\\S`[RbRZmXqSkgZQj (1087) in=40b,setxattr YgloTThONOYj[cntZofPOaP^O_bsWg]fd (1823) in=33b,setxattr ejbfqsafT_htneSabdatV`Qp^RS`f[nN``fN\\ebj (807) in=40b,setxattr Ya\\[Sha^pYao_k[dboRNsfOYcXoQtQRfV[XWcTQj (947) in=40b,setxattr YNVc]ZTPiOoc`NXSYsd[NZljlaqNVlYlRnihjYdQ (1789) in=40b,setxattr _t]sstiO]naqp_Zar\\Vtqss]cmN]kYnT_NtflgVd (9120) in=40b,setxattr lXVOWmpdrdYlUkq]]nPfdVVotggPsOZ\\VmPemila (909) in=40b,setxattr slQjq^hbmpO^nVhe\\te\\SkYldVebmeasZVgapUqn (454) in=40b,setxattr n[ea[_dQdc][]\\bfthkjd\\aSbNS^Qe_ZX\\Wr_^^V (594) in=40b,setxattr nS]tbdRhgf]l`\\Tfrnkihlk`Zpgbn[PY`anPZnhr (1682) in=40b,setxattr rpSrTfUUP^cVp_\\d]tNQnmXf[Np\\eO_YeldoslW_ (946) in=40b,setxattr VePmePiYj[hSVdatcprSk\\m]]tXhjbke]jZZYRQS (1681) in=40b,setxattr or[\\eoSZ^ZtXlcRqVTioQ[kZUnqnW[\\ (1823) in=31b,setxattr SS\\`X_`sgfdYjRdpNccP`gcPcRp^eSYXXOOs\\dN[ (1821) in=40b,setxattr obshrdURd^[OaVZZiSTRdPe^cqqgmSO[^Spi^Sgh (1821) in=40b,setxattr oUcnoPgXaRjdRatSbsPd[Ts]f\\pgbOl_Qk^R^Nfq (12280) in=40b,setxattr Xsgptojf]Obssrjj[_NfoopYpXifZhOd\\_]gWoXe (5052) in=40b] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.235876+0000", 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.319204961000001, 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.407899966, 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 529 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.579 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.235876+0000", 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.235876+0000", 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.235879+0000", 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3129999999999999e-06 2024-09-16T04:42:30.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.236042+0000", 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000162802 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.236074+0000", 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2305000000000003e-05 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.236096+0000", 2024-09-16T04:42:30.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1667000000000001e-05 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.236236+0000", 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000140739 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.236251+0000", 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4158999999999999e-05 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.342620+0000", 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10636910199999999 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.342989+0000", 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00036934700000000001 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.345604+0000", 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026144100000000002 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.349070+0000", 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0034668339999999998 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.643624+0000", 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29455326500000001 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.643776+0000", 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015202299999999999 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:577 3.es0 3:7777e846:::smithi16530038-17:head [append 0~491520] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:30.585 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.313373+0000", 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.241707323, 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41150533299999997, 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 577 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.313373+0000", 2024-09-16T04:42:30.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.313373+0000", 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.313376+0000", 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9730000000000002e-06 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.314319+0000", 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00094271899999999996 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.314355+0000", 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5589000000000001e-05 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.314376+0000", 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1897000000000001e-05 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.317116+0000", 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027390719999999999 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.317387+0000", 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027174000000000002 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.321727+0000", 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0043394319999999998 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.328638+0000", 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0069117520000000002 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.724750+0000", 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39611145599999997 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.724879+0000", 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000128703 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.591 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:578 3.es0 3:7777e846:::smithi16530038-17:head [append 0~589824] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.329745+0000", 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.225335464, 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39546170600000002, 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.592 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 578 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.329745+0000", 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.329745+0000", 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.329748+0000", 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6790000000000001e-06 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.330714+0000", 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00096573600000000002 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.330755+0000", 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0988999999999997e-05 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.330781+0000", 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6135999999999998e-05 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.330884+0000", 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000103354 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334464+0000", 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0035796339999999999 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.337572+0000", 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031087279999999998 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.342371+0000", 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0047990589999999996 2024-09-16T04:42:30.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.725095+0000", 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38272395100000001 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.725207+0000", 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011144 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:579 3.es0 3:7777e846:::smithi16530038-17:head [append 0~557056] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.333995+0000", 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.221085996999999, 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39148887900000001, 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 579 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.333995+0000", 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.333995+0000", 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.333998+0000", 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.027e-06 2024-09-16T04:42:30.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339182+0000", 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.005184307 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339205+0000", 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3393000000000001e-05 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339221+0000", 2024-09-16T04:42:30.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6141999999999999e-05 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339327+0000", 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000105523 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339555+0000", 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022766799999999999 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.341206+0000", 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016508860000000001 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.442767+0000", 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10156145599999999 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.725380+0000", 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28261286600000002 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.725484+0000", 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010361099999999999 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:581 3.es0 3:7777e846:::smithi16530038-17:head [setxattr _header (57) in=7b] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.339328+0000", 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.215752687, 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38706460500000001, 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.603 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 581 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339328+0000", 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339328+0000", 2024-09-16T04:42:30.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339329+0000", 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5199999999999995e-07 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339334+0000", 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.848e-06 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339345+0000", 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.081e-05 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339355+0000", 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0215e-05 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.445325+0000", 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105970469 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.445489+0000", 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016344499999999999 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446343+0000", 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00085430499999999997 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.447382+0000", 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010387199999999999 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.726228+0000", 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.278846602 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.726393+0000", 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000164239 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.608 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:582 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.215711927999999, 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69089629399999997, 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 582 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339370+0000", 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339374+0000", 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0799999999999999e-06 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339387+0000", 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3464e-05 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339414+0000", 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7223e-05 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446804+0000", 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.107389545 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446957+0000", 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000153389 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727239+0000", 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28028220199999998 2024-09-16T04:42:30.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727462+0000", 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022226099999999999 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.030265+0000", 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280332199999999 2024-09-16T04:42:30.613 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:613 3.es0 3:7777e846:::smithi16530038-17:head [append 0~589824] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.1948614400000004, 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55445315100000003, 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.614 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 613 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360223+0000", 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3730000000000001e-06 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361140+0000", 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00091743300000000005 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361178+0000", 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7691999999999998e-05 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361197+0000", 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.891e-05 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361351+0000", 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015419300000000001 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:30.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361371+0000", 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9726000000000001e-05 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366548+0000", 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051778730000000004 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366714+0000", 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000165329 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.369378+0000", 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026643159999999999 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.372406+0000", 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003027891 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914520+0000", 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54211425499999999 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914672+0000", 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015216000000000001 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:614 3.es0 3:7777e846:::smithi16530038-17:head [append 0~540672] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.1908883939999999, 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55075342599999999, 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.620 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 614 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364194+0000", 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.26e-06 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365041+0000", 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00084621499999999999 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365073+0000", 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2719000000000003e-05 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465237+0000", 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100163671 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465392+0000", 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015528300000000001 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465730+0000", 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000338005 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.467540+0000", 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001809583 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.469612+0000", 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020719760000000001 2024-09-16T04:42:30.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914847+0000", 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.44523504899999999 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914946+0000", 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8665000000000005e-05 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:615 3.es0 3:7777e846:::smithi16530038-17:head [append 0~622592] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.465323+0000", 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.0897575809999998, 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.44987216000000002, 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 615 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465323+0000", 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465323+0000", 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465326+0000", 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4029999999999999e-06 2024-09-16T04:42:30.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.466487+0000", 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001160454 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.466514+0000", 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6665999999999998e-05 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566617+0000", 2024-09-16T04:42:30.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100103912 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.574023+0000", 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0074054919999999996 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.574328+0000", 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0003054 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.576405+0000", 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020762459999999999 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579325+0000", 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029206340000000001 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.915098+0000", 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.33577316200000001 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.915195+0000", 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6791000000000002e-05 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:616 3.es0 3:7777e846:::smithi16530038-17:head [setxattr _header (57) in=7b] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566649+0000", 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.9884312360000003, 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50685167900000005, 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.631 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 616 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566649+0000", 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566649+0000", 2024-09-16T04:42:30.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566652+0000", 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1600000000000001e-06 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566663+0000", 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1326e-05 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566681+0000", 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8496e-05 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566693+0000", 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1143e-05 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.577966+0000", 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011273350999999999 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.578144+0000", 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017772299999999999 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.578890+0000", 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00074591299999999996 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579957+0000", 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010670860000000001 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.073307+0000", 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49335071800000002 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.073501+0000", 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019376300000000001 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:617 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:30.636 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.988374265, 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71993709900000002, 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 617 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:30.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566707+0000", 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566713+0000", 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.784e-06 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566725+0000", 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.173e-05 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566732+0000", 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6860000000000006e-06 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579439+0000", 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012706935000000001 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579759+0000", 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032009600000000002 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074376+0000", 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49461618299999999 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074520+0000", 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014481800000000001 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.286643+0000", 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212123017 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.641 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:692 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~688128] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.405376+0000", 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.1497041729999999, 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39004968099999998, 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.642 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 692 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.405376+0000", 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.405376+0000", 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.405379+0000", 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7e-06 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.406895+0000", 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015163450000000001 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.406925+0000", 2024-09-16T04:42:30.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9941e-05 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.406941+0000", 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5665999999999999e-05 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.485002+0000", 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.078060472000000006 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.485313+0000", 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031099099999999999 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.489154+0000", 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0038417500000000001 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.497630+0000", 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0084753959999999996 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.795218+0000", 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29758783799999999 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.795426+0000", 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020858199999999999 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:693 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~507904] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:30.647 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.413535+0000", 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.1415453010000001, 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50096031799999996, 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 693 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.413535+0000", 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.413535+0000", 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.413538+0000", 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6299999999999998e-06 2024-09-16T04:42:30.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.414473+0000", 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00093524999999999999 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.414501+0000", 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8028999999999999e-05 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.414517+0000", 2024-09-16T04:42:30.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5469e-05 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.492165+0000", 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077648791999999994 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.492366+0000", 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020003299999999999 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.494023+0000", 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016577479999999999 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502358+0000", 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0083349870000000003 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.914240+0000", 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41188197999999998 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.914496+0000", 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025539999999999997 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:694 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~655360] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.423435+0000", 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.1316455480000001, 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.491904444, 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.653 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 694 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.423435+0000", 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.423435+0000", 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.423437+0000", 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2929999999999999e-06 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424595+0000", 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011580270000000001 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424628+0000", 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2498e-05 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424643+0000", 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5271999999999999e-05 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.497750+0000", 2024-09-16T04:42:30.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073106666000000001 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.497889+0000", 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000139612 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.499106+0000", 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012162690000000001 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502368+0000", 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003261917 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915109+0000", 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41274159799999999 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915340+0000", 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000230292 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.658 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:695 3.10s0 3:0fc9379d:::smithi16530038-35:head [setxattr _header (57) in=7b] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.424670+0000", 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.130410776, 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49132234299999999, 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.659 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 695 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424670+0000", 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424670+0000", 2024-09-16T04:42:30.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424671+0000", 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1170000000000001e-06 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424677+0000", 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6400000000000002e-06 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424688+0000", 2024-09-16T04:42:30.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1299e-05 2024-09-16T04:42:30.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424696+0000", 2024-09-16T04:42:30.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4670000000000004e-06 2024-09-16T04:42:30.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.500062+0000", 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.075365550000000003 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.500342+0000", 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000279826 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.501803+0000", 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014609149999999999 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.503030+0000", 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001227349 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915728+0000", 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41269838599999997 2024-09-16T04:42:30.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915992+0000", 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026379399999999998 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:696 3.10s0 3:0fc9379d:::smithi16530038-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.1303720310000001, 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69507003199999995, 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.666 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 696 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424713+0000", 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5599999999999998e-06 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424724+0000", 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0906e-05 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424732+0000", 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6499999999999996e-06 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502418+0000", 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077686693000000001 2024-09-16T04:42:30.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502530+0000", 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000111644 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917065+0000", 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41453470199999998 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917391+0000", 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032674700000000001 2024-09-16T04:42:30.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.119779+0000", 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202387337 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:709 3.21s0 3:8583b9f2:::smithi16530038-39:head [copy-from ver 2] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.660672+0000", 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.894408333, 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.504571824, 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 709 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660672+0000", 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660672+0000", 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660677+0000", 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7269999999999998e-06 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660693+0000", 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6276000000000001e-05 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660731+0000", 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7840999999999999e-05 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:30.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660755+0000", 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4249000000000001e-05 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660932+0000", 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000176868 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.661230+0000", 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00029770099999999999 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.820627+0000", 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15939671699999999 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.913914+0000", 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.093287503999999993 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.165124+0000", 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.25120952899999999 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:30.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.165244+0000", 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120412 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:30.677 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:30.679 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:42:30.679 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:30.675+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:30.779 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:30.775+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:30.811 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:30.811 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:30.811 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:30.812 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:30.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:42:30.880 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:30.875+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:30.906 INFO:tasks.rados.rados.0.smithi165.stdout:286: finishing rollback tid 2 to smithi16530038-13 2024-09-16T04:42:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 59 (ObjNum 86 snap 14 seq_num 86) dirty exists 2024-09-16T04:42:30.949 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:30.949 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:30.950 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:30.950 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:30.950 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:30.963 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:42:31.010 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:31.007+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:31.082 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:31.079+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:31.091 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.677818336000001, 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20191694600000001, 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409856+0000", 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409917+0000", 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1023999999999997e-05 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409958+0000", 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0945000000000002e-05 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510088+0000", 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100130017 2024-09-16T04:42:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510247+0000", 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015866599999999999 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510287+0000", 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0330999999999997e-05 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611689+0000", 2024-09-16T04:42:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101401328 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611771+0000", 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2582000000000003e-05 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s6 44/37 ECSubWrite(tid=11, reqid=osd.5.0:11, at_version=44'8, trim_to=0'0, roll_forward_to=42'6))", 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 36.969573580000002, 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10484916599999999, 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118099+0000", 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118100+0000", 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9599999999999998e-07 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118155+0000", 2024-09-16T04:42:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4663999999999997e-05 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118199+0000", 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4861000000000003e-05 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118225+0000", 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5415999999999999e-05 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118379+0000", 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015390299999999999 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.118406+0000", 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7317999999999998e-05 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222865+0000", 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10445916099999999 2024-09-16T04:42:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.222948+0000", 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3047000000000006e-05 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 58/32 ECSubWrite(tid=118, reqid=client.4728.0:458, at_version=58'27, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.405078+0000", 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.682594405, 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103927176, 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405078+0000", 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405078+0000", 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405080+0000", 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9300000000000002e-06 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405351+0000", 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027029099999999998 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405381+0000", 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0571999999999998e-05 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405398+0000", 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6595999999999998e-05 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405533+0000", 2024-09-16T04:42:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013564600000000001 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.405555+0000", 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2079e-05 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.508926+0000", 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103370634 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.509005+0000", 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9427999999999999e-05 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.616268503000001, 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246694400000001, 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471405+0000", 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471443+0000", 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7605000000000003e-05 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471463+0000", 2024-09-16T04:42:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0452999999999999e-05 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.571565+0000", 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100102105 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671828+0000", 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026227 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671853+0000", 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5826e-05 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673802+0000", 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019481730000000001 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673871+0000", 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9624999999999998e-05 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 64/32 ECSubWrite(tid=56, reqid=client.4728.0:507, at_version=64'36, trim_to=0'0, roll_forward_to=63'34))", 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:14.692061+0000", 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.395611897999999, 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10500321, 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692061+0000", 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692061+0000", 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692062+0000", 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5859999999999999e-06 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692658+0000", 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00059599100000000003 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692684+0000", 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6046000000000001e-05 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.792795+0000", 2024-09-16T04:42:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10011104699999999 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.792888+0000", 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2812000000000005e-05 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.792903+0000", 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4477999999999999e-05 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.797015+0000", 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0041121250000000003 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.797064+0000", 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9125000000000001e-05 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 69/67 ECSubWrite(tid=147, reqid=client.4728.0:574, at_version=69'46, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.406141+0000", 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.681532159, 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.119323211, 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406141+0000", 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406141+0000", 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406142+0000", 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406184+0000", 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1356000000000002e-05 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406213+0000", 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9592999999999999e-05 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406230+0000", 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6129000000000001e-05 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.522241+0000", 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.116011266 2024-09-16T04:42:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.522268+0000", 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6837000000000001e-05 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.525400+0000", 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031320190000000002 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.525464+0000", 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4115000000000005e-05 2024-09-16T04:42:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 69/67 ECSubWrite(tid=148, reqid=client.4728.0:575, at_version=69'47, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.409732+0000", 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.677940502, 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.1165219, 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.409732+0000", 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.409732+0000", 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.409734+0000", 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.697e-06 2024-09-16T04:42:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.409769+0000", 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4656000000000002e-05 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.409801+0000", 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2842000000000002e-05 2024-09-16T04:42:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.409818+0000", 2024-09-16T04:42:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6633000000000002e-05 2024-09-16T04:42:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.524591+0000", 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.114772675 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.524610+0000", 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8995999999999999e-05 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.526203+0000", 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015928629999999999 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.526254+0000", 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1538e-05 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 69/67 ECSubWrite(tid=149, reqid=client.4728.0:576, at_version=69'48, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.576727641, 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30202165600000003, 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510947+0000", 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.95e-06 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510983+0000", 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6198999999999997e-05 2024-09-16T04:42:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.511014+0000", 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1112000000000003e-05 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.611120+0000", 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100105729 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811521+0000", 2024-09-16T04:42:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040100599999999 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811553+0000", 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1958999999999997e-05 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812884+0000", 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013312700000000001 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812967+0000", 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2430999999999997e-05 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s6 70/55 ECSubWrite(tid=34, reqid=osd.5.0:33, at_version=0'0, trim_to=0'0, roll_forward_to=70'21))", 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.461173+0000", 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.626499665000001, 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10824056899999999, 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.461173+0000", 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.461173+0000", 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.461175+0000", 2024-09-16T04:42:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8670000000000001e-06 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.461202+0000", 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7599000000000001e-05 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.461247+0000", 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4771999999999998e-05 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.461276+0000", 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8255999999999998e-05 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.567627+0000", 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106351721 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.567653+0000", 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5296e-05 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.569324+0000", 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001671854 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.569414+0000", 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9203999999999995e-05 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 70/67 ECSubWrite(tid=155, reqid=osd.4.0:153, at_version=0'0, trim_to=0'0, roll_forward_to=70'52))", 2024-09-16T04:42:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.521807141, 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137054300000001, 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565867+0000", 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.84e-06 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565893+0000", 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5715999999999999e-05 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565933+0000", 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0358999999999998e-05 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666040+0000", 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010693499999999 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666191+0000", 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015013299999999999 2024-09-16T04:42:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666251+0000", 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0615000000000002e-05 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767150+0000", 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100898584 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767236+0000", 2024-09-16T04:42:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6360999999999995e-05 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=59, reqid=osd.8.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.5847173220000004, 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26917078799999999, 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502956+0000", 2024-09-16T04:42:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3e-07 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502980+0000", 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3802e-05 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.503017+0000", 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7082999999999997e-05 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603116+0000", 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098623 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603238+0000", 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012259700000000001 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603255+0000", 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6762999999999999e-05 2024-09-16T04:42:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772084+0000", 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16882903799999999 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772126+0000", 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2151999999999999e-05 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=97, reqid=client.4728.0:613, at_version=74'68, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.520728503, 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20513274000000001, 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566945+0000", 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.03e-07 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567199+0000", 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025406099999999998 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567228+0000", 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9589000000000001e-05 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567244+0000", 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6027000000000001e-05 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567424+0000", 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000180006 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567456+0000", 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1251999999999998e-05 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772018+0000", 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20456203000000001 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772077+0000", 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9372e-05 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=98, reqid=client.4728.0:614, at_version=74'69, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.5203879840000001, 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.205026973, 2024-09-16T04:42:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567286+0000", 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567509+0000", 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022361500000000001 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567536+0000", 2024-09-16T04:42:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7063000000000002e-05 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567555+0000", 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669312+0000", 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101756135 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669353+0000", 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1069000000000003e-05 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772248+0000", 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102895556 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772312+0000", 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3559000000000006e-05 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=60, reqid=client.4728.0:618, at_version=74'51, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.605149+0000", 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.4825235939999999, 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.167020959, 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605149+0000", 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605149+0000", 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605151+0000", 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4729999999999999e-06 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605196+0000", 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5735999999999997e-05 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605235+0000", 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8209e-05 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605259+0000", 2024-09-16T04:42:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4085999999999999e-05 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605435+0000", 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017617300000000001 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605455+0000", 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9882e-05 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772138+0000", 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16668382400000001 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772170+0000", 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1575999999999997e-05 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=61, reqid=client.4728.0:619, at_version=74'52, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.607451+0000", 2024-09-16T04:42:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.4802216760000002, 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.164789196, 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607451+0000", 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607451+0000", 2024-09-16T04:42:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607453+0000", 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.306e-06 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607501+0000", 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7691999999999997e-05 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607549+0000", 2024-09-16T04:42:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8222e-05 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607572+0000", 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3240000000000001e-05 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607736+0000", 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016364299999999999 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607769+0000", 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3036e-05 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772175+0000", 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.164405632 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772240+0000", 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5425000000000005e-05 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 79/37 ECSubWrite(tid=59, reqid=osd.15.0:59, at_version=79'48, trim_to=0'0, roll_forward_to=78'46))", 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.60897994, 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.119802273, 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2300000000000002e-07 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478737+0000", 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4024999999999999e-05 2024-09-16T04:42:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478764+0000", 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6838999999999998e-05 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478783+0000", 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8702000000000002e-05 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.596719+0000", 2024-09-16T04:42:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.117936155 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.596737+0000", 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7915999999999999e-05 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.598372+0000", 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001635372 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.598495+0000", 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000122841 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 79/37 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=79'50, trim_to=0'0, roll_forward_to=78'46))", 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.481265+0000", 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.6064076439999999, 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.118018944, 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481265+0000", 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481265+0000", 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481267+0000", 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6729999999999999e-06 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481303+0000", 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6358e-05 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481333+0000", 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9869000000000002e-05 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481349+0000", 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6039e-05 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.597457+0000", 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.116108302 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.597481+0000", 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4148999999999999e-05 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.599238+0000", 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017566159999999999 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.599284+0000", 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5938e-05 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=74, reqid=client.4728.0:707, at_version=0'0, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.71539886799999997, 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202706724, 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372276+0000", 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.782e-06 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372297+0000", 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0907000000000002e-05 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372320+0000", 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3759000000000001e-05 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372333+0000", 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2719e-05 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472559+0000", 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10022637399999999 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472570+0000", 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0351e-05 2024-09-16T04:42:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574932+0000", 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10236213700000001 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574981+0000", 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8695e-05 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=75, reqid=client.4728.0:708, at_version=79'59, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.71298046699999995, 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20032509600000001, 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374694+0000", 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374738+0000", 2024-09-16T04:42:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3016000000000003e-05 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374782+0000", 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4709999999999997e-05 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374813+0000", 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0700000000000001e-05 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473059+0000", 2024-09-16T04:42:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098245669999999993 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473071+0000", 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.24e-05 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574989+0000", 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10191757 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575017+0000", 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8773e-05 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 80/67 ECSubWrite(tid=114, reqid=osd.2.0:114, at_version=80'75, trim_to=0'0, roll_forward_to=79'73))", 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.693829582, 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18121299299999999, 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393844+0000", 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5000000000000003e-07 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393900+0000", 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6436e-05 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393925+0000", 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4703999999999999e-05 2024-09-16T04:42:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393948+0000", 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3173e-05 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473572+0000", 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079623997000000002 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473583+0000", 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1327e-05 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575021+0000", 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101437639 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575056+0000", 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5166999999999997e-05 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.185 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:31.186 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:42:31.187 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:31.183+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:31.287 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:31.283+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:31.355 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:31.356 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:31.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:42:31.388 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:31.383+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:31.393 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T04:42:31.398 INFO:tasks.rados.rados.0.smithi165.stdout:281: done (7 left) 2024-09-16T04:42:31.398 INFO:tasks.rados.rados.0.smithi165.stdout:282: done (6 left) 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:283: done (5 left) 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:284: done (4 left) 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:286: done (3 left) 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:288: done (2 left) 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:289: done (1 left) 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:291: done (0 left) 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 16 current snap is 21 2024-09-16T04:42:31.399 INFO:tasks.rados.rados.0.smithi165.stdout:292: seq_num 112 ranges {1523712=720896,2244608=802816,3047424=360448} 2024-09-16T04:42:31.420 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T04:42:31.435 INFO:tasks.rados.rados.0.smithi165.stdout:292: writing smithi16530038-16 from 1523712 to 2244608 tid 1 2024-09-16T04:42:31.447 INFO:tasks.rados.rados.0.smithi165.stdout:292: writing smithi16530038-16 from 2244608 to 3047424 tid 2 2024-09-16T04:42:31.452 INFO:tasks.rados.rados.0.smithi165.stdout:292: writing smithi16530038-16 from 3047424 to 3407872 tid 3 2024-09-16T04:42:31.452 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 18 current snap is 21 2024-09-16T04:42:31.452 INFO:tasks.rados.rados.0.smithi165.stdout:293: seq_num 113 ranges {0=737280,737280=557056,1294336=606208} 2024-09-16T04:42:31.461 INFO:tasks.rados.rados.0.smithi165.stdout:293: writing smithi16530038-18 from 0 to 737280 tid 1 2024-09-16T04:42:31.467 INFO:tasks.rados.rados.0.smithi165.stdout:293: writing smithi16530038-18 from 737280 to 1294336 tid 2 2024-09-16T04:42:31.473 INFO:tasks.rados.rados.0.smithi165.stdout:293: writing smithi16530038-18 from 1294336 to 1900544 tid 3 2024-09-16T04:42:31.473 INFO:tasks.rados.rados.0.smithi165.stdout:294: setattr oid 45 current snap is 21 2024-09-16T04:42:31.475 INFO:tasks.rados.rados.0.smithi165.stdout:295: snap_remove snap 9 2024-09-16T04:42:31.489 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:31.483+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:31.530 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:31.530 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:31.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:42:31.590 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:31.587+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:31.668 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=17, reqid=client.4728.0:139, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.577956735999997, 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.203994169, 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086853+0000", 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6000000000000001e-06 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087506+0000", 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00065279900000000002 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087552+0000", 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5840999999999997e-05 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087581+0000", 2024-09-16T04:42:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9354999999999999e-05 2024-09-16T04:42:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187852+0000", 2024-09-16T04:42:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100270242 2024-09-16T04:42:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187866+0000", 2024-09-16T04:42:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4286e-05 2024-09-16T04:42:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290791+0000", 2024-09-16T04:42:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102925166 2024-09-16T04:42:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290845+0000", 2024-09-16T04:42:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3879999999999999e-05 2024-09-16T04:42:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=18, reqid=client.4728.0:140, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.575271331000003, 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27450581499999999, 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089538+0000", 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0370000000000001e-06 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089605+0000", 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6268999999999999e-05 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089663+0000", 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8257000000000001e-05 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.189764+0000", 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100101525 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290404+0000", 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10063934500000001 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290428+0000", 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4782000000000001e-05 2024-09-16T04:42:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363982+0000", 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073553303 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364042+0000", 2024-09-16T04:42:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0297000000000002e-05 2024-09-16T04:42:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 38/32 ECSubWrite(tid=32, reqid=client.4728.0:209, at_version=38'17, trim_to=0'0, roll_forward_to=36'15))", 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 43.189746507999999, 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202405108, 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:42:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475061+0000", 2024-09-16T04:42:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475063+0000", 2024-09-16T04:42:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.906e-06 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475102+0000", 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9195e-05 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475132+0000", 2024-09-16T04:42:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9861000000000001e-05 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.475148+0000", 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.611e-05 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.575413+0000", 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026479000000001 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.575431+0000", 2024-09-16T04:42:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7912000000000001e-05 2024-09-16T04:42:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677395+0000", 2024-09-16T04:42:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101964155 2024-09-16T04:42:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.677466+0000", 2024-09-16T04:42:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1179000000000004e-05 2024-09-16T04:42:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/32 ECSubWrite(tid=62, reqid=osd.7.0:61, at_version=0'0, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T04:42:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 37.157749529999997, 2024-09-16T04:42:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20295831, 2024-09-16T04:42:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507061+0000", 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5059999999999998e-06 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507088+0000", 2024-09-16T04:42:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7764999999999999e-05 2024-09-16T04:42:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507122+0000", 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3671999999999999e-05 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607262+0000", 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100139588 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707564+0000", 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100302577 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707595+0000", 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0482000000000001e-05 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.709942+0000", 2024-09-16T04:42:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002347671 2024-09-16T04:42:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.710016+0000", 2024-09-16T04:42:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4048999999999995e-05 2024-09-16T04:42:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/37 ECSubWrite(tid=84, reqid=client.4728.0:380, at_version=49'15, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.460045313999998, 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20242805, 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204766+0000", 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.523e-06 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205263+0000", 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00049713700000000003 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205312+0000", 2024-09-16T04:42:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8908e-05 2024-09-16T04:42:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205338+0000", 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6602e-05 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207188+0000", 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018498200000000001 2024-09-16T04:42:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207215+0000", 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6322e-05 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407111+0000", 2024-09-16T04:42:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.199896194 2024-09-16T04:42:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407190+0000", 2024-09-16T04:42:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9543999999999996e-05 2024-09-16T04:42:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/37 ECSubWrite(tid=85, reqid=client.4728.0:381, at_version=49'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.458177838000001, 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20298417399999999, 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206631+0000", 2024-09-16T04:42:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206948+0000", 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031706300000000002 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206984+0000", 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6256e-05 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207008+0000", 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3726e-05 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.405710+0000", 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19870283399999999 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.405729+0000", 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8116000000000001e-05 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.409550+0000", 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0038217580000000002 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.409614+0000", 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3621000000000006e-05 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 58/37 ECSubWrite(tid=116, reqid=client.4728.0:454, at_version=58'26, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:42:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 23.260819344000002, 2024-09-16T04:42:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23856791699999999, 2024-09-16T04:42:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403990+0000", 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9769999999999999e-06 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404088+0000", 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7986999999999995e-05 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404119+0000", 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0975000000000001e-05 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404135+0000", 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5874000000000001e-05 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.540429+0000", 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13629391299999999 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.540454+0000", 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5075999999999998e-05 2024-09-16T04:42:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642480+0000", 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102026085 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642556+0000", 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6030000000000002e-05 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 58/37 ECSubWrite(tid=117, reqid=client.4728.0:455, at_version=58'27, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 23.260644609, 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23846093700000001, 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404164+0000", 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9299999999999998e-07 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404183+0000", 2024-09-16T04:42:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9066e-05 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404196+0000", 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2999999999999999e-05 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404205+0000", 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4229999999999994e-06 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.541363+0000", 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.137157952 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.541386+0000", 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3028999999999999e-05 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642560+0000", 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101174403 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642624+0000", 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3471000000000003e-05 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 58/32 ECSubWrite(tid=116, reqid=client.4728.0:462, at_version=58'44, trim_to=0'0, roll_forward_to=57'42))", 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.828199688000002, 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20377977999999999, 2024-09-16T04:42:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836610+0000", 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836817+0000", 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020725999999999999 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836849+0000", 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2149e-05 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.937028+0000", 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100178188 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.037317+0000", 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100289105 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.037342+0000", 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5812e-05 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.040305+0000", 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002962059 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.040388+0000", 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3153999999999997e-05 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.727 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.412090804999998, 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.396784737, 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.46e-07 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253567+0000", 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00085024700000000005 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253594+0000", 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6529e-05 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253616+0000", 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2373000000000001e-05 2024-09-16T04:42:31.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448095+0000", 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19447869800000001 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448103+0000", 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9379999999999992e-06 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649474+0000", 2024-09-16T04:42:31.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137053299999999 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649502+0000", 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7973000000000001e-05 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:31.732 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.406392651000001, 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.391121679, 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:31.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258417+0000", 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.722e-06 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258658+0000", 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024151700000000001 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258682+0000", 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3526000000000001e-05 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258695+0000", 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3225e-05 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548388+0000", 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28969289100000001 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548393+0000", 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2970000000000003e-06 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649523+0000", 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101130161 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649537+0000", 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3339999999999999e-05 2024-09-16T04:42:31.736 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.403926010999999, 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38866693400000002, 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260883+0000", 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0750000000000001e-06 2024-09-16T04:42:31.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260979+0000", 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6552999999999995e-05 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260998+0000", 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.261010+0000", 2024-09-16T04:42:31.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1817999999999999e-05 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548799+0000", 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28778851599999999 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548839+0000", 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0371999999999999e-05 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649539+0000", 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100699682 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649549+0000", 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8050000000000001e-06 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:31.741 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.402287713, 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38775422500000001, 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:31.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262521+0000", 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.009e-06 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262542+0000", 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0981000000000001e-05 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262560+0000", 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8224e-05 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262572+0000", 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2155e-05 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649287+0000", 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38671449499999999 2024-09-16T04:42:31.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649295+0000", 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8329999999999994e-06 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650254+0000", 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00095943799999999996 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650274+0000", 2024-09-16T04:42:31.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0089999999999999e-05 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpRead(3.17s2 65/53 ECSubRead(tid=26, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.344820+0000", 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.319987289, 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20354055099999999, 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344820+0000", 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344820+0000", 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344823+0000", 2024-09-16T04:42:31.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2740000000000002e-06 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344842+0000", 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8936e-05 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344880+0000", 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8801999999999999e-05 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344900+0000", 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0066000000000001e-05 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548281+0000", 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20338055399999999 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548361+0000", 2024-09-16T04:42:31.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9919000000000005e-05 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.359637+0000", 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.305170347000001, 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28247715000000001, 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.750 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359637+0000", 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359637+0000", 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359640+0000", 2024-09-16T04:42:31.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5730000000000002e-06 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359648+0000", 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0069999999999997e-06 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359714+0000", 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6128000000000001e-05 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359735+0000", 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0579999999999999e-05 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642075+0000", 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28234038299999997 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642114+0000", 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9478999999999999e-05 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.753 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.366437+0000", 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.298370745, 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27593029899999999, 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366437+0000", 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366437+0000", 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366439+0000", 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9740000000000001e-06 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366448+0000", 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6229999999999992e-06 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366476+0000", 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7966e-05 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366498+0000", 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1246e-05 2024-09-16T04:42:31.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642331+0000", 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.275833571 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642367+0000", 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5918999999999999e-05 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.757 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.241767347, 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.32087775200000002, 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:31.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423042+0000", 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.77e-06 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423051+0000", 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8100000000000004e-06 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423081+0000", 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0626000000000001e-05 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523175+0000", 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100093245 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742603+0000", 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21942863300000001 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.743918+0000", 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001314668 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.761 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 70/67 ECSubWrite(tid=154, reqid=osd.4.0:151, at_version=0'0, trim_to=0'0, roll_forward_to=70'33))", 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.523286+0000", 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.141521331, 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.22189425900000001, 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523286+0000", 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523286+0000", 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523288+0000", 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2529999999999999e-06 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523305+0000", 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7286e-05 2024-09-16T04:42:31.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523331+0000", 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6389000000000001e-05 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523352+0000", 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0619999999999999e-05 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742851+0000", 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21949913700000001 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742882+0000", 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1092e-05 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.745111+0000", 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002228561 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.745181+0000", 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9920999999999997e-05 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 79/67 ECSubWrite(tid=188, reqid=client.4728.0:711, at_version=79'67, trim_to=0'0, roll_forward_to=79'65))", 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.714432+0000", 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.95037602, 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.303689612, 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.766 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714432+0000", 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714432+0000", 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714434+0000", 2024-09-16T04:42:31.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.95e-06 2024-09-16T04:42:31.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714485+0000", 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1632999999999998e-05 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714523+0000", 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7954999999999998e-05 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814634+0000", 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100110872 2024-09-16T04:42:31.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915010+0000", 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100376302 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915028+0000", 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7558999999999999e-05 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018009+0000", 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102981302 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018121+0000", 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112039 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 79/67 ECSubWrite(tid=189, reqid=client.4728.0:712, at_version=79'68, trim_to=0'0, roll_forward_to=79'65))", 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.814716+0000", 2024-09-16T04:42:31.771 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.850091191, 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20348661600000001, 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814716+0000", 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814716+0000", 2024-09-16T04:42:31.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814718+0000", 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814775+0000", 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7293999999999997e-05 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:31.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814807+0000", 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1894999999999999e-05 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.914910+0000", 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100103152 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915929+0000", 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010187760000000001 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915960+0000", 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0375999999999999e-05 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018136+0000", 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10217612600000001 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018203+0000", 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7306999999999998e-05 2024-09-16T04:42:31.775 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.776 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:31.776 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:31.776 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:31.777 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:42:31.778 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:31.775+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:31.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:31.878 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:31.871+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:31.951 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:31.952 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:31.952 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:31.952 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:31.966 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:42:31.978 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:31.975+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:31.984 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:32.006 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:32.027 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:32.027 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:32.079 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:32.071+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:32.087 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:32.087 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:32.088 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:32.088 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:32.088 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:32.093 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:32.101 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:42:32.118 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:32.118 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:32.146 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:32.146 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:32.173 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:32.173 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:32.180 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:32.175+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:32.200 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:32.201 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:32.224 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:32.228 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:32.228 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:32.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:32.228 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:157 3.9s0 3:9de467d5:::smithi16530038-27:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.740035511999999, 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51218579099999995, 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.229 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 157 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484010+0000", 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0329999999999998e-06 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484272+0000", 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026256800000000002 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484296+0000", 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3509000000000001e-05 2024-09-16T04:42:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484308+0000", 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2699e-05 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584144+0000", 2024-09-16T04:42:32.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099835667000000003 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584304+0000", 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016037700000000001 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.585781+0000", 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001476455 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589205+0000", 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0034239499999999998 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996077+0000", 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40687225300000002 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996193+0000", 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011628 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:158 3.9s0 3:9de467d5:::smithi16530038-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.739711984000003, 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51808159200000004, 2024-09-16T04:42:32.235 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 158 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:32.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484332+0000", 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484339+0000", 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8630000000000002e-06 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484355+0000", 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6473e-05 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484364+0000", 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8950000000000004e-06 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.586917+0000", 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102553035 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.587177+0000", 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026005499999999998 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.588403+0000", 2024-09-16T04:42:32.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001225152 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589911+0000", 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015086260000000001 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002165+0000", 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41225355800000002 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002413+0000", 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024783200000000002 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.9s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.739643174999998, 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.62506688499999996, 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.241 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484401+0000", 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484405+0000", 2024-09-16T04:42:32.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.827e-06 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484416+0000", 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1609e-05 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484425+0000", 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790928+0000", 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30650327300000002 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.791067+0000", 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013860200000000001 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003346+0000", 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212279577 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003611+0000", 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026456899999999998 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109440+0000", 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105828857 2024-09-16T04:42:32.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109467+0000", 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6900999999999999e-05 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.248 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.19s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.116690568000003, 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.976424197, 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.249 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107354+0000", 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9829999999999999e-06 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107362+0000", 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6129999999999998e-06 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107388+0000", 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5675000000000001e-05 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207482+0000", 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009412299999999 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207644+0000", 2024-09-16T04:42:32.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000161741 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207649+0000", 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0359999999999997e-06 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082558+0000", 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8749096240000001 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082784+0000", 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022614900000000001 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.083777+0000", 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099225300000000002 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:178 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:32.254 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 44.145929142, 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1069151189999999, 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 178 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078116+0000", 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:42:32.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078127+0000", 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.062e-05 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078151+0000", 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4304999999999999e-05 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078175+0000", 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4006e-05 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776496+0000", 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69832100500000005 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776503+0000", 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.533e-06 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185014+0000", 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40851148700000001 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185029+0000", 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4780000000000001e-05 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:179 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:32.259 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 44.145833779999997, 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1068670979999999, 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 179 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.260 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078211+0000", 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.44e-06 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078220+0000", 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078244+0000", 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3867999999999999e-05 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078260+0000", 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6203000000000001e-05 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780436+0000", 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70217526500000005 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780441+0000", 2024-09-16T04:42:32.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4620000000000004e-06 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185067+0000", 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40462590799999998 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185076+0000", 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3219999999999999e-06 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:180 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 44.145760723999999, 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.106828843, 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 180 2024-09-16T04:42:32.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078284+0000", 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078291+0000", 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.657e-06 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078310+0000", 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078325+0000", 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4772e-05 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981190+0000", 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90286589500000003 2024-09-16T04:42:32.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981194+0000", 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0790000000000002e-06 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185100+0000", 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20390586399999999 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185111+0000", 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0794e-05 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:185 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:32.270 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 44.125444745000003, 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98955797199999995, 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:32.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098600+0000", 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098611+0000", 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0423e-05 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098642+0000", 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1532000000000002e-05 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098658+0000", 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5996000000000001e-05 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683726+0000", 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58506727400000003 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683731+0000", 2024-09-16T04:42:32.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088142+0000", 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40441113000000001 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088156+0000", 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4148000000000001e-05 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:186 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 44.125356689999997, 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.089963279, 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.276 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 186 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098687+0000", 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.063e-06 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098693+0000", 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6230000000000002e-06 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098705+0000", 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2019e-05 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198804+0000", 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098513 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683799+0000", 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48499579100000001 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683801+0000", 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0889999999999998e-06 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188641+0000", 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50483997300000005 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188650+0000", 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.208e-06 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.282 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:187 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 44.025207287000001, 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98988477799999997, 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 187 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198838+0000", 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.86e-06 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198846+0000", 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0059999999999992e-06 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198864+0000", 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8468999999999999e-05 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198876+0000", 2024-09-16T04:42:32.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1443e-05 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783957+0000", 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58508126900000001 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783962+0000", 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7230000000000001e-06 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.286 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188710+0000", 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40474714899999997 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188721+0000", 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0859000000000001e-05 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:583 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:32.287 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.889721331000001, 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2114807170000002, 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 583 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.288 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334325+0000", 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6969999999999999e-06 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334340+0000", 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4508999999999999e-05 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334388+0000", 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8016000000000002e-05 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334413+0000", 2024-09-16T04:42:32.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5284999999999999e-05 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828480+0000", 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406639099999999 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828485+0000", 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3430000000000003e-06 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.769267+0000", 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.94078222199999995 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.771858+0000", 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025913479999999998 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174292+0000", 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40243307299999997 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174565+0000", 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027297299999999998 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.176259+0000", 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001694161 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281556+0000", 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10529764799999999 2024-09-16T04:42:32.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545252+0000", 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.263695658 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545802+0000", 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00055039300000000004 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:584 3.1as0 3:5f4281cc:::smithi16530038-45:head [rollback 11] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.889586868, 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2123693160000002, 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 584 2024-09-16T04:42:32.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334458+0000", 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.663e-06 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334465+0000", 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2239999999999998e-06 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334485+0000", 2024-09-16T04:42:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9925999999999999e-05 2024-09-16T04:42:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334500+0000", 2024-09-16T04:42:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5449000000000001e-05 2024-09-16T04:42:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828570+0000", 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49407003700000002 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828576+0000", 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.93e-06 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872096+0000", 2024-09-16T04:42:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.043520019 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872408+0000", 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031130200000000001 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177318+0000", 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30491066900000002 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177579+0000", 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026102699999999999 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.178857+0000", 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001277395 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281984+0000", 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10312769099999999 2024-09-16T04:42:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546082+0000", 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26409753200000002 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546825+0000", 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00074345200000000002 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:585 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.889522996, 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3685253789999998, 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 585 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334521+0000", 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4169999999999999e-06 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334529+0000", 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8560000000000007e-06 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334547+0000", 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7813e-05 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334561+0000", 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3803e-05 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828622+0000", 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406153000000003 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828624+0000", 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.277e-06 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873077+0000", 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.044453106 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873349+0000", 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027208400000000002 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279758+0000", 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40640956900000003 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279965+0000", 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000206028 2024-09-16T04:42:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281051+0000", 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010864399999999999 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.282237+0000", 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011862839999999999 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.701843+0000", 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41960623800000002 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.703045+0000", 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001201934 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.as4 70/67 ECSubWrite(tid=83, reqid=osd.3.0:83, at_version=70'33, trim_to=0'0, roll_forward_to=68'29))", 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.360901+0000", 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.863142066, 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50776604400000003, 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.360901+0000", 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.360901+0000", 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.360902+0000", 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.387e-06 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.360951+0000", 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9066000000000001e-05 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.360977+0000", 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5225e-05 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.360996+0000", 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9386999999999999e-05 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.560992+0000", 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19999566899999999 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.561006+0000", 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3886999999999999e-05 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.868604+0000", 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.307597958 2024-09-16T04:42:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.868667+0000", 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3465000000000003e-05 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:630 3.19s0 3:99864b0e:::smithi16530038-47:head [read 0~1818624,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.143403+0000", 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.0806403910000002, 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.414582589, 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 630 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143403+0000", 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143403+0000", 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143406+0000", 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.96e-06 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143419+0000", 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3529999999999999e-05 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143453+0000", 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3563999999999999e-05 2024-09-16T04:42:32.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143469+0000", 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6288999999999999e-05 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143682+0000", 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00021284200000000001 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:32.321 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143700+0000", 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7745999999999999e-05 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.245628+0000", 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101928054 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.245973+0000", 2024-09-16T04:42:32.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00034557199999999999 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.557985+0000", 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31201203199999999 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.323 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:631 3.19s0 3:99864b0e:::smithi16530038-47:head [read 0~1818624,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.143507+0000", 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.0805364060000002, 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41767522499999998, 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.324 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 631 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143507+0000", 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143507+0000", 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143508+0000", 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.043e-06 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143513+0000", 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7660000000000004e-06 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143528+0000", 2024-09-16T04:42:32.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.438e-05 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143538+0000", 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0614999999999999e-05 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143802+0000", 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026360000000000001 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143819+0000", 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6999000000000001e-05 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.246847+0000", 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10302842600000001 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.247259+0000", 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000411488 2024-09-16T04:42:32.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.561182+0000", 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.313922908 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:632 3.19s0 3:99864b0e:::smithi16530038-47:head [read 0~1818624,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.143585+0000", 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.0804579570000001, 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.42041208499999999, 2024-09-16T04:42:32.329 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 632 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143585+0000", 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143585+0000", 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143586+0000", 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.077e-06 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143591+0000", 2024-09-16T04:42:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8319999999999996e-06 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143605+0000", 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3857e-05 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.143615+0000", 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0468999999999999e-05 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.146514+0000", 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0028985080000000002 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.146539+0000", 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5378e-05 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.249289+0000", 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10275010800000001 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.250510+0000", 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001220247 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.563997+0000", 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.313487609 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.334 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:633 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~622592] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.157242+0000", 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.06680098, 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.42842454699999999, 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 633 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.335 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.157242+0000", 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.157242+0000", 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.157244+0000", 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9400000000000001e-06 2024-09-16T04:42:32.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.158574+0000", 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013302609999999999 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.158604+0000", 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9705000000000001e-05 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.158620+0000", 2024-09-16T04:42:32.337 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6019999999999999e-05 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.170260+0000", 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011640217 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.170445+0000", 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000184737 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.338 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.172640+0000", 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002195139 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.378681+0000", 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206040848 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.585493+0000", 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206812096 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.339 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.585667+0000", 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017358400000000001 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:634 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~442368] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.164632+0000", 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.0594113780000001, 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.42142702100000001, 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 634 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.164632+0000", 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.164632+0000", 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.164634+0000", 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2469999999999998e-06 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.165371+0000", 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00073671000000000001 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.165394+0000", 2024-09-16T04:42:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3635e-05 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.165406+0000", 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2017e-05 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.274782+0000", 2024-09-16T04:42:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10937535700000001 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.275012+0000", 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022995700000000001 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.276586+0000", 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015741399999999999 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.479050+0000", 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246438 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.585926+0000", 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106875942 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.586059+0000", 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000132636 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:635 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~524288] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.0506633870000002, 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71243645600000005, 2024-09-16T04:42:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 635 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173382+0000", 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174349+0000", 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00096718499999999996 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174408+0000", 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9036000000000002e-05 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174432+0000", 2024-09-16T04:42:32.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4176e-05 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579526+0000", 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40509398299999999 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579719+0000", 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019240699999999999 2024-09-16T04:42:32.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.581395+0000", 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016763769999999999 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.586083+0000", 2024-09-16T04:42:32.352 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0046879950000000004 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885565+0000", 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29948223000000002 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:32.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885816+0000", 2024-09-16T04:42:32.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025068399999999999 2024-09-16T04:42:32.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.354 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:32.354 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:32.354 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:32.355 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:32.356 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:32.356 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:32.357 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:32.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:42:32.358 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:32.351+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:32.358 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:32.358 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:32.358 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:32.359 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:32.368 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:32.368 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:32.396 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:32.397 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:32.405 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T04:42:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:295: done (3 left) 2024-09-16T04:42:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:296: setattr oid 20 current snap is 21 2024-09-16T04:42:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 49 (ObjNum 106 snap 20 seq_num 106) dirty exists 2024-09-16T04:42:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 1 to smithi16530038-16 2024-09-16T04:42:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 2 to smithi16530038-16 2024-09-16T04:42:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 3 to smithi16530038-16 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: finishing write tid 1 to smithi16530038-18 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: finishing write tid 2 to smithi16530038-18 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: finishing write tid 3 to smithi16530038-18 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: finishing write tid 4 to smithi16530038-18 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: finishing write tid 5 to smithi16530038-18 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: oid 18 updating version 0 to 122 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: oid 18 version 122 is already newer than 120 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: oid 18 version 122 is already newer than 121 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: oid 18 version 122 is already newer than 119 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 122 (ObjNum 113 snap 21 seq_num 113) dirty exists 2024-09-16T04:42:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:293: left oid 18 (ObjNum 113 snap 21 seq_num 113) 2024-09-16T04:42:32.411 INFO:tasks.rados.rados.0.smithi165.stdout:293: done (3 left) 2024-09-16T04:42:32.411 INFO:tasks.rados.rados.0.smithi165.stdout:294: done (2 left) 2024-09-16T04:42:32.411 INFO:tasks.rados.rados.0.smithi165.stdout:297: snap_create 2024-09-16T04:42:32.417 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:42:32.429 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:32.429 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:32.440 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T04:42:32.457 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:32.451+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:32.457 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 4 to smithi16530038-16 2024-09-16T04:42:32.498 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:32.499 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:32.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:42:32.557 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:32.555+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:32.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 72 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-09-16T04:42:32.658 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:32.655+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:32.678 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:32.679 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:32.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:42:32.730 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 5 to smithi16530038-16 2024-09-16T04:42:32.730 INFO:tasks.rados.rados.0.smithi165.stdout:292: oid 16 updating version 0 to 63 2024-09-16T04:42:32.730 INFO:tasks.rados.rados.0.smithi165.stdout:292: oid 16 version 63 is already newer than 60 2024-09-16T04:42:32.730 INFO:tasks.rados.rados.0.smithi165.stdout:292: oid 16 version 63 is already newer than 62 2024-09-16T04:42:32.730 INFO:tasks.rados.rados.0.smithi165.stdout:292: oid 16 version 63 is already newer than 61 2024-09-16T04:42:32.730 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 63 (ObjNum 112 snap 21 seq_num 112) dirty exists 2024-09-16T04:42:32.731 INFO:tasks.rados.rados.0.smithi165.stdout:292: left oid 16 (ObjNum 112 snap 21 seq_num 112) 2024-09-16T04:42:32.760 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:32.755+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:32.824 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:32.824 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:32.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:42:32.860 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:32.855+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:32.961 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:32.955+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:33.009 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:33.009 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:33.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:33.062 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:33.059+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:33.128 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:33.123+0000 7ff1d2c53640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap b after trimming was started 2024-09-16T04:42:33.146 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:33.146 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:33.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:33.162 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:33.159+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:33.263 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:33.259+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:33.286 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:33.286 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:33.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:33.364 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:33.359+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:33.410 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T04:42:33.410 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:42:33.414 INFO:tasks.rados.rados.0.smithi165.stdout:292: done (2 left) 2024-09-16T04:42:33.414 INFO:tasks.rados.rados.0.smithi165.stdout:296: done (1 left) 2024-09-16T04:42:33.414 INFO:tasks.rados.rados.0.smithi165.stdout:297: done (0 left) 2024-09-16T04:42:33.414 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 33 current snap is 22 2024-09-16T04:42:33.414 INFO:tasks.rados.rados.0.smithi165.stdout:298: seq_num 114 ranges {1523712=753664,2277376=688128,2965504=475136} 2024-09-16T04:42:33.428 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:33.428 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:33.438 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T04:42:33.441 INFO:tasks.rados.rados.0.smithi165.stdout:298: writing smithi16530038-33 from 1523712 to 2277376 tid 1 2024-09-16T04:42:33.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:33.449 INFO:tasks.rados.rados.0.smithi165.stdout:298: writing smithi16530038-33 from 2277376 to 2965504 tid 2 2024-09-16T04:42:33.455 INFO:tasks.rados.rados.0.smithi165.stdout:298: writing smithi16530038-33 from 2965504 to 3440640 tid 3 2024-09-16T04:42:33.455 INFO:tasks.rados.rados.0.smithi165.stdout:299: copy_from oid 18 from oid 13 current snap is 22 2024-09-16T04:42:33.455 INFO:tasks.rados.rados.0.smithi165.stdout:300: snap_create 2024-09-16T04:42:33.483 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:33.479+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:33.566 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:33.563+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:33.568 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:33.568 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:33.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:33.667 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:33.663+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:33.720 INFO:tasks.rados.rados.0.smithi165.stdout:298: finishing write tid 1 to smithi16530038-33 2024-09-16T04:42:33.720 INFO:tasks.rados.rados.0.smithi165.stdout:298: finishing write tid 2 to smithi16530038-33 2024-09-16T04:42:33.748 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:33.748 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:33.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:42:33.810 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:33.807+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:33.822 INFO:tasks.rados.rados.0.smithi165.stdout:298: finishing write tid 3 to smithi16530038-33 2024-09-16T04:42:33.840 INFO:tasks.rados.rados.0.smithi165.stdout:299: finishing copy_from racing read to smithi16530038-18 2024-09-16T04:42:33.841 INFO:tasks.rados.rados.0.smithi165.stdout:298: finishing write tid 4 to smithi16530038-33 2024-09-16T04:42:33.869 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:33.863+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:33.883 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:33.883 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:33.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:42:33.942 INFO:tasks.rados.rados.0.smithi165.stdout:299: finishing copy_from to smithi16530038-18 2024-09-16T04:42:33.943 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 59 (ObjNum 86 snap 14 seq_num 86) dirty exists 2024-09-16T04:42:33.970 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:33.963+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:34.043 INFO:tasks.rados.rados.0.smithi165.stdout:298: finishing write tid 5 to smithi16530038-33 2024-09-16T04:42:34.044 INFO:tasks.rados.rados.0.smithi165.stdout:298: oid 33 updating version 0 to 129 2024-09-16T04:42:34.044 INFO:tasks.rados.rados.0.smithi165.stdout:298: oid 33 updating version 129 to 131 2024-09-16T04:42:34.044 INFO:tasks.rados.rados.0.smithi165.stdout:298: oid 33 version 131 is already newer than 130 2024-09-16T04:42:34.044 INFO:tasks.rados.rados.0.smithi165.stdout:298: oid 33 version 131 is already newer than 128 2024-09-16T04:42:34.044 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 131 (ObjNum 114 snap 22 seq_num 114) dirty exists 2024-09-16T04:42:34.044 INFO:tasks.rados.rados.0.smithi165.stdout:298: left oid 33 (ObjNum 114 snap 22 seq_num 114) 2024-09-16T04:42:34.064 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:34.065 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:34.071 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:34.067+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:34.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:42:34.171 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:34.167+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:34.243 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:34.243 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:34.257 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:34.272 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:34.267+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:34.373 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:34.367+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:34.392 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:34.392 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:34.407 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:34.430 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T04:42:34.434 INFO:tasks.rados.rados.0.smithi165.stdout:298: done (2 left) 2024-09-16T04:42:34.434 INFO:tasks.rados.rados.0.smithi165.stdout:299: done (1 left) 2024-09-16T04:42:34.434 INFO:tasks.rados.rados.0.smithi165.stdout:300: done (0 left) 2024-09-16T04:42:34.434 INFO:tasks.rados.rados.0.smithi165.stdout:301: snap_create 2024-09-16T04:42:34.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T04:42:34.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:34.498 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:34.495+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:34.537 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:34.538 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:34.555 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:34.575 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:34.571+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:34.639 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:34.640 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:34.662 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:34.662 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:34.677 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:34.671+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:34.696 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:34.697 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:34.717 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:34.730 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:34.730 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:34.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s4 36/32 ECSubWrite(tid=8, reqid=client.4728.0:109, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.353508669999997, 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201785732, 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374507+0000", 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374649+0000", 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014185599999999999 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374672+0000", 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3277000000000001e-05 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374685+0000", 2024-09-16T04:42:34.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2797e-05 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574940+0000", 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20025501600000001 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574956+0000", 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5936000000000001e-05 2024-09-16T04:42:34.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576258+0000", 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001302388 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576291+0000", 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.735 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 36/32 ECSubWrite(tid=4, reqid=client.4728.0:112, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.154402343000001, 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20173359499999999, 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573614+0000", 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.114e-06 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574099+0000", 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048458300000000002 2024-09-16T04:42:34.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574141+0000", 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2551e-05 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574168+0000", 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6899000000000001e-05 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674350+0000", 2024-09-16T04:42:34.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100181464 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674362+0000", 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2138e-05 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775305+0000", 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100943171 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775345+0000", 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0674999999999999e-05 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:164 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.830479908999997, 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40676020400000001, 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.741 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 164 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:34.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897538+0000", 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0500000000000002e-06 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897554+0000", 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5886e-05 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897591+0000", 2024-09-16T04:42:34.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6899e-05 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897617+0000", 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5922e-05 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897727+0000", 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011011200000000001 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897835+0000", 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107889 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304263+0000", 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40642834999999999 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304295+0000", 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1096000000000002e-05 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.745 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:165 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.830340221, 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20425574899999999, 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:34.746 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 165 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897676+0000", 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897685+0000", 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.183e-06 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897709+0000", 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4216e-05 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997840+0000", 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131417 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098015+0000", 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100174894 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098124+0000", 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108714 2024-09-16T04:42:34.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101885+0000", 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037611459999999999 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101930+0000", 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4675999999999997e-05 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.750 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.751 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:166 3.6s0 3.a17ee16 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:34.751 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:34.751 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.730123958, 2024-09-16T04:42:34.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.306539124, 2024-09-16T04:42:34.751 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 166 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997892+0000", 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997905+0000", 2024-09-16T04:42:34.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2583e-05 2024-09-16T04:42:34.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997926+0000", 2024-09-16T04:42:34.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.101e-05 2024-09-16T04:42:34.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098022+0000", 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100096478 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304408+0000", 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20638583099999999 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304429+0000", 2024-09-16T04:42:34.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1132e-05 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.16s0 42/41 ECSubRead(tid=6, to_read={3:6877e850:::smithi16530038-5:head=0,20480,0}, subchunks={3:6877e850:::smithi16530038-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.419790892000002, 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20348143299999999, 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308260+0000", 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6318e-05 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308312+0000", 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2611999999999997e-05 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511485+0000", 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20317243800000001 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511705+0000", 2024-09-16T04:42:34.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220065 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.419115058999999, 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20303517200000001, 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308901+0000", 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308970+0000", 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9240999999999997e-05 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309005+0000", 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4314999999999999e-05 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309023+0000", 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8879999999999999e-05 2024-09-16T04:42:34.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511826+0000", 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202802133 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511934+0000", 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108756 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.418880266999999, 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20302847199999999, 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309135+0000", 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309242+0000", 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106959 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309260+0000", 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309270+0000", 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0339e-05 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512036+0000", 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20276591199999999 2024-09-16T04:42:34.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512162+0000", 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126675 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.14s4 43/41 ECSubWrite(tid=30, reqid=client.4728.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 41.388737865000003, 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20195774699999999, 2024-09-16T04:42:34.768 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339277+0000", 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.61e-07 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339289+0000", 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1533e-05 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339303+0000", 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4002000000000001e-05 2024-09-16T04:42:34.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439391+0000", 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10008851100000001 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439604+0000", 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301100000000001 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439617+0000", 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3076e-05 2024-09-16T04:42:34.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541138+0000", 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10152072299999999 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541234+0000", 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6130000000000003e-05 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 53/32 ECSubWrite(tid=89, reqid=client.4728.0:426, at_version=53'29, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.793819847999998, 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20199651099999999, 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934195+0000", 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934336+0000", 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014033399999999999 2024-09-16T04:42:34.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934357+0000", 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1769000000000001e-05 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034519+0000", 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016164599999999 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034685+0000", 2024-09-16T04:42:34.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000166312 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034712+0000", 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6959999999999999e-05 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136138+0000", 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101425772 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136191+0000", 2024-09-16T04:42:34.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2868000000000003e-05 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=115, reqid=client.4728.0:457, at_version=58'26, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.428658688999999, 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.240164188, 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.777 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5699999999999999e-07 2024-09-16T04:42:34.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299490+0000", 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013428299999999999 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299510+0000", 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437999999999999e-05 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299523+0000", 2024-09-16T04:42:34.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3132000000000001e-05 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436652+0000", 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13712892300000001 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436659+0000", 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4410000000000004e-06 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539460+0000", 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102800512 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539520+0000", 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0201999999999997e-05 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=122, reqid=client.4728.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=58'29))", 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:34.781 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.915532878, 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20212734700000001, 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812484+0000", 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3700000000000002e-06 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812511+0000", 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.775e-05 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812550+0000", 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812567+0000", 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7186000000000001e-05 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812770+0000", 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202862 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812799+0000", 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8716000000000001e-05 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014571+0000", 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201771586 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014609+0000", 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8124999999999998e-05 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.47522111, 2024-09-16T04:42:34.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.296481516, 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252794+0000", 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253432+0000", 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063874400000000001 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253463+0000", 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0052999999999998e-05 2024-09-16T04:42:34.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253487+0000", 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4335999999999999e-05 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344896+0000", 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.091408780999999995 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344925+0000", 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9490000000000001e-05 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549182+0000", 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.204257051 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549275+0000", 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2464000000000001e-05 2024-09-16T04:42:34.790 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.469430253999999, 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29077158600000003, 2024-09-16T04:42:34.791 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258586+0000", 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258848+0000", 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026226500000000002 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258888+0000", 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0395999999999997e-05 2024-09-16T04:42:34.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258917+0000", 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8371999999999999e-05 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.346992+0000", 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.088075353999999995 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.347031+0000", 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:42:34.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549288+0000", 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20225760800000001 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549356+0000", 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7222000000000001e-05 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.466563520000001, 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.288032389, 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.796 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261453+0000", 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.79e-06 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261566+0000", 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000113801 2024-09-16T04:42:34.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261597+0000", 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0772999999999998e-05 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261613+0000", 2024-09-16T04:42:34.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:42:34.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348618+0000", 2024-09-16T04:42:34.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.087004695000000007 2024-09-16T04:42:34.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348634+0000", 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6246e-05 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549401+0000", 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200766994 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549483+0000", 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2150999999999993e-05 2024-09-16T04:42:34.800 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.464868500000001, 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28676506400000001, 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263148+0000", 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.81e-06 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263180+0000", 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2019e-05 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263210+0000", 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0429999999999998e-05 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263226+0000", 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6093e-05 2024-09-16T04:42:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548818+0000", 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28559150300000002 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548834+0000", 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6804e-05 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549811+0000", 2024-09-16T04:42:34.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00097657100000000008 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549911+0000", 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9834000000000007e-05 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds4 70/67 ECSubWrite(tid=146, reqid=osd.7.0:146, at_version=70'118, trim_to=0'0, roll_forward_to=70'116))", 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:20.558888+0000", 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.169125822, 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20169158400000001, 2024-09-16T04:42:34.805 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558888+0000", 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558888+0000", 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558890+0000", 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1519999999999999e-06 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558927+0000", 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7248e-05 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.558940+0000", 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3352e-05 2024-09-16T04:42:34.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.659049+0000", 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100108354 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.759316+0000", 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10026745099999999 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.759349+0000", 2024-09-16T04:42:34.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2923000000000003e-05 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.760523+0000", 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011739350000000001 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.760580+0000", 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7169000000000001e-05 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/67 ECSubWrite(tid=159, reqid=client.4728.0:622, at_version=74'134, trim_to=0'0, roll_forward_to=72'132))", 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.254417773, 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.205798497, 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473597+0000", 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3999999999999998e-07 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473620+0000", 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3383999999999998e-05 2024-09-16T04:42:34.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473630+0000", 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473639+0000", 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9799999999999993e-06 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676469+0000", 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202829168 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676503+0000", 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4239999999999997e-05 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679328+0000", 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028252860000000002 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679395+0000", 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6773000000000006e-05 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.21s5,QREQUEST,e:74,from:2(0),reservation_nonce:2)", 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.7914488350000006, 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30070015300000003, 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.814 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936567+0000", 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.86e-06 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936576+0000", 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0559999999999994e-06 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936605+0000", 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8632999999999998e-05 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.036785+0000", 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10017992000000001 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237143+0000", 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20035802999999999 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237193+0000", 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0121999999999999e-05 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237266+0000", 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2532000000000003e-05 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.bs1 76/67 ECSubRead(tid=164, to_read={3:d388eed4:::smithi16530038-28:head=0,339968,0}, subchunks={3:d388eed4:::smithi16530038-28:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.4501076939999997, 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22646649499999999, 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:34.818 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:34.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277908+0000", 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7439999999999999e-06 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277925+0000", 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7084000000000001e-05 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277960+0000", 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4208999999999997e-05 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.820 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277978+0000", 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8219e-05 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504200+0000", 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22622223899999999 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504373+0000", 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000173 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.821 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.822 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:34.822 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:34.822 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:34.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:34.824 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:34.824 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:34.824 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:34.824 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:34.825 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:34.825 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:34.825 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:34.826 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:34.826 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:34.819+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:34.839 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:34.895 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:34.921 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:34.924 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:34.919+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:34.949 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:34.949 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:34.964 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:34.964 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:34.964 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:34.964 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:34.973 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:34.973 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:34.977 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:34.999 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:34.999 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:35.025 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:35.026 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:35.019+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:35.026 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:35.051 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:35.118 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:35.118 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:35.118 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:35.118 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:35.119 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:35.126 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:35.123+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:35.131 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:35.227 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:35.223+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:35.255 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:35.255 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:35.255 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:35.255 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:137 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.189188987999998, 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50711364199999998, 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 137 2024-09-16T04:42:35.256 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062314+0000", 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062318+0000", 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1710000000000004e-06 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062327+0000", 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0089999999999996e-06 2024-09-16T04:42:35.258 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162444+0000", 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011658 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163173+0000", 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072917800000000003 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163440+0000", 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026697899999999998 2024-09-16T04:42:35.259 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164004+0000", 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056357499999999997 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164775+0000", 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00077136499999999998 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.260 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569249+0000", 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.404473996 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569427+0000", 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177906 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:138 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:35.261 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.089020105000003, 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.71018989099999996, 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 138 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:35.262 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162484+0000", 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162491+0000", 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162507+0000", 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5141e-05 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162514+0000", 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8460000000000008e-06 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.264 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165370+0000", 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028555719999999998 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165438+0000", 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8466999999999993e-05 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570190+0000", 2024-09-16T04:42:35.265 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40475178899999997 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570271+0000", 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0437000000000003e-05 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.872672+0000", 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30240151100000001 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.266 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:181 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.159146839000002, 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.514197601, 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.267 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 181 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:35.268 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092358+0000", 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9500000000000001e-06 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093295+0000", 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00093678900000000005 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.269 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093330+0000", 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5039000000000001e-05 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093348+0000", 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7210999999999998e-05 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269509+0000", 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17616195900000001 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269804+0000", 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029419599999999999 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.271 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272744+0000", 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029401789999999998 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279866+0000", 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007122499 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606426+0000", 2024-09-16T04:42:35.272 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32655929 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606553+0000", 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000127489 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.273 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:182 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.152818394000001, 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50813697599999996, 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 182 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.274 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0260000000000002e-06 2024-09-16T04:42:35.275 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099161+0000", 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000474556 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099192+0000", 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1612000000000002e-05 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.276 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099207+0000", 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4796e-05 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275225+0000", 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17601765799999999 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275471+0000", 2024-09-16T04:42:35.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024600099999999998 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001437422 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280012+0000", 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031043899999999998 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606726+0000", 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32671404399999998 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606821+0000", 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4470999999999997e-05 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.279 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:183 3.bs0 3:d388eed4:::smithi16530038-28:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.152267985000002, 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60643071599999998, 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.280 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 183 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.281 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099235+0000", 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.017e-06 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.282 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099242+0000", 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4230000000000002e-06 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099256+0000", 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3936e-05 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099264+0000", 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278076+0000", 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17881208600000001 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278318+0000", 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024238100000000001 2024-09-16T04:42:35.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279368+0000", 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010492209999999999 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281553+0000", 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021853739999999999 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705572+0000", 2024-09-16T04:42:35.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.42401889100000001 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705665+0000", 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3173999999999999e-05 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:191 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:35.286 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.150407829999999, 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68460226999999996, 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 191 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:35.287 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101096+0000", 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-06 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101104+0000", 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1240000000000005e-06 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101122+0000", 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8045000000000001e-05 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101130+0000", 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2670000000000006e-06 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280859+0000", 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179728363 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280864+0000", 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7370000000000003e-06 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.405994+0000", 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.12512986500000001 2024-09-16T04:42:35.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.406243+0000", 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024852700000000001 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.785697+0000", 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.37945393900000002 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:192 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.150355611999998, 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68518017200000003, 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 192 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.292 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:35.293 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101154+0000", 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3300000000000004e-06 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101160+0000", 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.348e-06 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101164+0000", 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2899999999999996e-06 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281071+0000", 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179906749 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281074+0000", 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.976e-06 2024-09-16T04:42:35.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506776+0000", 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22570168299999999 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506893+0000", 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011752300000000001 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786327+0000", 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27943377000000003 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:201 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.150222524999997, 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68324074999999995, 2024-09-16T04:42:35.297 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 201 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.299 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101282+0000", 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9750000000000001e-06 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101287+0000", 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.651e-06 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.201419+0000", 2024-09-16T04:42:35.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10013264099999999 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482351+0000", 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28093166899999999 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482482+0000", 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013094199999999999 2024-09-16T04:42:35.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.784521+0000", 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30203853899999999 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:372 3.1s0 3:8583b9f2:::smithi16530038-39:3 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:35.302 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.070220227, 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.58438020099999999, 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 372 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.303 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.304 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181283+0000", 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1100000000000005e-07 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181288+0000", 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4739999999999996e-06 2024-09-16T04:42:35.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181300+0000", 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1266e-05 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181308+0000", 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5669999999999995e-06 2024-09-16T04:42:35.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181619+0000", 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031025600000000002 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181627+0000", 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3329999999999997e-06 2024-09-16T04:42:35.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288322+0000", 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669511499999999 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288524+0000", 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202511 2024-09-16T04:42:35.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.765662+0000", 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.47713786800000002 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:452 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.992548371000002, 2024-09-16T04:42:35.309 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57389822599999996, 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 452 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.310 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258957+0000", 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9510000000000002e-06 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260061+0000", 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011037410000000001 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260099+0000", 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8401e-05 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260123+0000", 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3808999999999999e-05 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260226+0000", 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010314000000000001 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260310+0000", 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3652000000000005e-05 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.261217+0000", 2024-09-16T04:42:35.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090691199999999995 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.262686+0000", 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014689 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832721+0000", 2024-09-16T04:42:35.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57003547600000004 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832852+0000", 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013124400000000001 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.316 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:453 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~622592] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.981870496999999, 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56351327799999995, 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.317 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 453 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269634+0000", 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2069999999999998e-06 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270168+0000", 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053375500000000002 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270185+0000", 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7231000000000002e-05 2024-09-16T04:42:35.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270194+0000", 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4349999999999997e-06 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270266+0000", 2024-09-16T04:42:35.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2702999999999998e-05 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270366+0000", 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9485e-05 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271233+0000", 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00086726900000000005 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374033+0000", 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10280007300000001 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833038+0000", 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45900526899999999 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833145+0000", 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106851 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.322 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:454 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~147456] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.979605716999998, 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56224230100000006, 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 454 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.323 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271898+0000", 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.364e-06 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272075+0000", 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017664100000000001 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272088+0000", 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3192e-05 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272097+0000", 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4720000000000001e-06 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372245+0000", 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10014781 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372332+0000", 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7258999999999999e-05 2024-09-16T04:42:35.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372854+0000", 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052174600000000002 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374728+0000", 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018736219999999999 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834034+0000", 2024-09-16T04:42:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45930626099999999 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834139+0000", 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000104934 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:455 3.1s0 3:803446a7:::smithi16530038-31:head [setxattr _header (57) in=7b] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.979381354000001, 2024-09-16T04:42:35.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.66155386500000002, 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 455 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272122+0000", 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272127+0000", 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5369999999999997e-06 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272135+0000", 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272141+0000", 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5999999999999997e-06 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373376+0000", 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101234779 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373445+0000", 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9507999999999999e-05 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373742+0000", 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029695500000000001 2024-09-16T04:42:35.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374731+0000", 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098912400000000008 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933530+0000", 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.55879828600000003 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933675+0000", 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145353 2024-09-16T04:42:35.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:456 3.1s0 3:803446a7:::smithi16530038-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.979350171, 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.76406687699999998, 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 456 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272153+0000", 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0100000000000005e-07 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272155+0000", 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.638e-06 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272165+0000", 2024-09-16T04:42:35.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1579999999999994e-06 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272171+0000", 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373963+0000", 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10179178799999999 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:35.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374024+0000", 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0791999999999997e-05 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934462+0000", 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56043857600000002 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934706+0000", 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024387000000000001 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.036219+0000", 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10151300000000001 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:458 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.948081107, 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50818626099999997, 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.339 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 458 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303426+0000", 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5870000000000003e-06 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303961+0000", 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053495599999999995 2024-09-16T04:42:35.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303975+0000", 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3835999999999999e-05 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404043+0000", 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100068845 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404113+0000", 2024-09-16T04:42:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9350999999999994e-05 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404242+0000", 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012877400000000001 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405312+0000", 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001070812 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407819+0000", 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002506131 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811485+0000", 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40366653800000002 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811608+0000", 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012243099999999999 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:461 3.bs0 3:d1897791:::smithi16530038-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.847348404999998, 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.61182177000000004, 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 461 2024-09-16T04:42:35.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:35.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6399999999999998e-07 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404155+0000", 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2550000000000001e-06 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404160+0000", 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0890000000000002e-06 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404162+0000", 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8399999999999999e-06 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507951+0000", 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103788373 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.508127+0000", 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017614900000000001 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813061+0000", 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30493453100000001 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813333+0000", 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027161699999999998 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.015976+0000", 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20264265200000001 2024-09-16T04:42:35.349 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:574 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.947739417999999, 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50935247100000003, 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 574 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303765+0000", 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.827e-06 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303772+0000", 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6210000000000004e-06 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303788+0000", 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5760000000000002e-05 2024-09-16T04:42:35.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.403903+0000", 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100114464 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404144+0000", 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024159600000000001 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404641+0000", 2024-09-16T04:42:35.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000496598 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.406835+0000", 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021939149999999998 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511811+0000", 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.104976429 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.812951+0000", 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30113980800000001 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.813115+0000", 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016445300000000001 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:576 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.847473752999999, 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51026414200000003, 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 576 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:35.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6199999999999998e-07 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404032+0000", 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6280000000000001e-06 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404038+0000", 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6200000000000001e-06 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404043+0000", 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5020000000000003e-06 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.510735+0000", 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669203400000001 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.510837+0000", 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101847 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511366+0000", 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052975600000000004 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.512713+0000", 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001346168 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.914111+0000", 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40139793200000001 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.914293+0000", 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018219300000000001 2024-09-16T04:42:35.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:644 3.bs0 3:d388eed4:::smithi16530038-28:head [read 0~1359872,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.0741917129999994, 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51132577199999996, 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.361 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 644 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7e-07 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177315+0000", 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.523e-06 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177322+0000", 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.968e-06 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177327+0000", 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.631e-06 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.458181+0000", 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28085389999999999 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.458352+0000", 2024-09-16T04:42:35.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017090399999999999 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.688636+0000", 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.23028437600000001 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:717 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~802816] snapc 16=[16,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.8032072590000001, 2024-09-16T04:42:35.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50523997499999995, 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 717 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:42:35.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448297+0000", 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2539999999999999e-06 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449278+0000", 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098078800000000002 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449311+0000", 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2768999999999997e-05 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449326+0000", 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5392000000000001e-05 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.537454+0000", 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.088127592000000005 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.537466+0000", 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2189e-05 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.544288+0000", 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0068214950000000003 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.544574+0000", 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000286113 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.547183+0000", 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026089809999999998 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.651536+0000", 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.104353801 2024-09-16T04:42:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.953403+0000", 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30186636999999999 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.953535+0000", 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013223099999999999 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.371 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:35.373 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:35.374 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:35.367+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:35.438 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T04:42:35.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:42:35.444 INFO:tasks.rados.rados.0.smithi165.stdout:301: done (0 left) 2024-09-16T04:42:35.445 INFO:tasks.rados.rados.0.smithi165.stdout:302: read oid 15 snap -1 2024-09-16T04:42:35.445 INFO:tasks.rados.rados.0.smithi165.stdout:302: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:35.445 INFO:tasks.rados.rados.0.smithi165.stdout:303: read oid 1 snap -1 2024-09-16T04:42:35.445 INFO:tasks.rados.rados.0.smithi165.stdout:303: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:35.445 INFO:tasks.rados.rados.0.smithi165.stdout:304: read oid 10 snap -1 2024-09-16T04:42:35.445 INFO:tasks.rados.rados.0.smithi165.stdout:304: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:35.445 INFO:tasks.rados.rados.0.smithi165.stdout:305: delete oid 49 current snap is 24 2024-09-16T04:42:35.465 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T04:42:35.474 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:35.471+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:35.553 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:35.553 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:35.553 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:35.553 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:35.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:42:35.577 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:35.571+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:35.653 INFO:tasks.rados.rados.0.smithi165.stdout:304: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:35.677 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:35.671+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:35.734 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:35.734 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:35.734 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:35.734 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:35.734 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:35.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:42:35.756 INFO:tasks.rados.rados.0.smithi165.stdout:302: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:35.778 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:35.775+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:35.850 INFO:tasks.rados.rados.0.smithi165.stdout:303: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:148 3.4s0 3:2b4d2b94:::smithi16530038-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.671622726999999, 2024-09-16T04:42:35.873 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.49811553600000003, 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 148 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198240+0000", 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3600000000000003e-07 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.875 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198243+0000", 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.281e-06 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198250+0000", 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7870000000000004e-06 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298324+0000", 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100073891 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298376+0000", 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2790000000000001e-05 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298437+0000", 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0550999999999997e-05 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298723+0000", 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000285741 2024-09-16T04:42:35.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.299721+0000", 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00099841899999999996 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696249+0000", 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39652776099999998 2024-09-16T04:42:35.878 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696354+0000", 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105679 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:149 3.4s0 3:2b4d2b94:::smithi16530038-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.571510248999999, 2024-09-16T04:42:35.879 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.59985384399999997, 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 149 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.880 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298354+0000", 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:42:35.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298362+0000", 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3319999999999992e-06 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298377+0000", 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5387999999999999e-05 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298388+0000", 2024-09-16T04:42:35.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0957e-05 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398555+0000", 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016630999999999 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398706+0000", 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015114699999999999 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696493+0000", 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29778684 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696643+0000", 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015093699999999999 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.898205+0000", 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20156174700000001 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:160 3.4s0 3:2d731f27:::smithi16530038-30:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.415880182999999, 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44291274600000002, 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 160 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453983+0000", 2024-09-16T04:42:35.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4729999999999999e-06 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453989+0000", 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1299999999999998e-06 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454005+0000", 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5880999999999999e-05 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.888 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454014+0000", 2024-09-16T04:42:35.888 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1079999999999999e-06 2024-09-16T04:42:35.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.888 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.888 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454090+0000", 2024-09-16T04:42:35.888 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6062999999999993e-05 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454166+0000", 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5772999999999994e-05 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454458+0000", 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029229599999999999 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.889 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.455106+0000", 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00064800899999999995 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896766+0000", 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44165951199999998 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896894+0000", 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000128501 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.890 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.14s0 42/41 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.037161466999997, 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60655789999999998, 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.891 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832701+0000", 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6400000000000001e-07 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.892 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832733+0000", 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2382000000000003e-05 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832757+0000", 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3691999999999999e-05 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.932887+0000", 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100130238 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.438947+0000", 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50605998399999996 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439258+0000", 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031074 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.894 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=56, reqid=osd.11.0:56, at_version=44'19, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 41.750967019000001, 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39966409400000003, 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.895 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6400000000000003e-07 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118922+0000", 2024-09-16T04:42:35.896 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6635000000000002e-05 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118941+0000", 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9707999999999998e-05 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118952+0000", 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.039e-05 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.897 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.513442+0000", 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39449013100000002 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.513459+0000", 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6748e-05 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.898 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.518526+0000", 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0050673970000000004 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.518559+0000", 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2620999999999998e-05 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.899 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=59, reqid=osd.11.0:59, at_version=44'21, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 41.748101576000003, 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.397602065, 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121760+0000", 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121761+0000", 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0699999999999996e-07 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121787+0000", 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6227000000000001e-05 2024-09-16T04:42:35.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121804+0000", 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6898999999999999e-05 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.121815+0000", 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1294000000000001e-05 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.514082+0000", 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39226631200000001 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.514105+0000", 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3366e-05 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.519327+0000", 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005222122 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.519362+0000", 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5037999999999999e-05 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.14.22:1 3.4s0 3:208a5e75:::smithi16530038-37:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:35.904 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 37.208936880000003, 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40385833399999999, 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660929+0000", 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7859999999999998e-06 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660942+0000", 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3526999999999999e-05 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660996+0000", 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3860999999999998e-05 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.761100+0000", 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100103686 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.857146+0000", 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.096046099999999995 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.857321+0000", 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000175465 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.064783+0000", 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.207461909 2024-09-16T04:42:35.908 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:368 3.14s0 3:28a14731:::smithi16530038-3:head [setxattr _header (55) in=7b] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.684820774000002, 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50583271100000005, 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.909 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 368 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.910 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185043+0000", 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.463e-06 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185054+0000", 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0356e-05 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185080+0000", 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6519000000000001e-05 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185097+0000", 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.717e-05 2024-09-16T04:42:35.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.188751+0000", 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036535249999999999 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.189019+0000", 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026812599999999999 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.190416+0000", 2024-09-16T04:42:35.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013973830000000001 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.389174+0000", 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19875738700000001 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690722+0000", 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30154792000000002 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690874+0000", 2024-09-16T04:42:35.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000151862 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:369 3.14s0 3:28a14731:::smithi16530038-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.684739057000002, 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.81195188399999996, 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.915 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 369 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185124+0000", 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.33e-06 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185130+0000", 2024-09-16T04:42:35.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9070000000000004e-06 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185154+0000", 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4059000000000001e-05 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185173+0000", 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8936e-05 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388708+0000", 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.203534774 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388994+0000", 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028647200000000002 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691636+0000", 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.302641681 2024-09-16T04:42:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691904+0000", 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026824200000000002 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.997074+0000", 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30517048299999999 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:618 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [append 0~0] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.500562497000001, 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.54505251200000004, 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 618 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369302+0000", 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6639999999999998e-06 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369315+0000", 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3416e-05 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369345+0000", 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9961e-05 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369359+0000", 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4107000000000001e-05 2024-09-16T04:42:35.925 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.604180+0000", 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.23482078300000001 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.604381+0000", 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020054599999999999 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.605620+0000", 2024-09-16T04:42:35.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012394699999999999 2024-09-16T04:42:35.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.608601+0000", 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029811059999999999 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914102+0000", 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.305500773 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914352+0000", 2024-09-16T04:42:35.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024968599999999999 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:619 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [rollback 11] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.500474988000001, 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.54536771900000003, 2024-09-16T04:42:35.929 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 619 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.930 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.931 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.68e-07 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369390+0000", 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.948e-06 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369399+0000", 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3499999999999997e-06 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369405+0000", 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.455e-06 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.606687+0000", 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.237281837 2024-09-16T04:42:35.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.606930+0000", 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024356099999999999 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.607700+0000", 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000769507 2024-09-16T04:42:35.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.810765+0000", 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20306539400000001 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914602+0000", 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103836502 2024-09-16T04:42:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914754+0000", 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000152597 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:620 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [append 0~0] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:42:35.936 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.500449229999999, 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.81742312399999995, 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 620 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369413+0000", 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9099999999999999e-07 2024-09-16T04:42:35.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369426+0000", 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.365e-05 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369435+0000", 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4700000000000002e-06 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.469519+0000", 2024-09-16T04:42:35.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100084346 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.912089+0000", 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44256948200000001 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.912270+0000", 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018106700000000001 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.913255+0000", 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098478400000000005 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914789+0000", 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015345210000000001 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.186759+0000", 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27197024199999997 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.186836+0000", 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6171e-05 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.942 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:706 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.861944+0000", 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.0079172969999997, 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40975806300000001, 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 706 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861944+0000", 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861944+0000", 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861946+0000", 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9099999999999999e-06 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861955+0000", 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1330000000000005e-06 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861979+0000", 2024-09-16T04:42:35.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3779000000000001e-05 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862007+0000", 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7628999999999999e-05 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862157+0000", 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015022200000000001 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862408+0000", 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025079200000000001 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.863488+0000", 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001080592 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.070476+0000", 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.206987744 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271563+0000", 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20108668199999999 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271702+0000", 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013957999999999999 2024-09-16T04:42:35.948 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:707 3.1es0 3:78f34faf:::smithi16530038-13:head [rollback 11] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.0078287640000001, 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51046276300000004, 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.949 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 707 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.950 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862034+0000", 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862038+0000", 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9160000000000003e-06 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.951 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862049+0000", 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1786e-05 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862057+0000", 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.966992+0000", 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10493451600000001 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.967212+0000", 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021991 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.968498+0000", 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012864350000000001 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271112+0000", 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30261409299999997 2024-09-16T04:42:35.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372370+0000", 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10125774899999999 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372496+0000", 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000125501 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.954 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:708 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.0077476699999997, 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.74368462499999999, 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:35.955 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 708 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.956 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7800000000000002e-07 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862116+0000", 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.08e-06 2024-09-16T04:42:35.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862122+0000", 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7930000000000003e-06 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862126+0000", 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2470000000000002e-06 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373611+0000", 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51148433299999996 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373943+0000", 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00033231500000000002 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.375178+0000", 2024-09-16T04:42:35.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001235367 2024-09-16T04:42:35.960 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.960 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:35.960 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.376541+0000", 2024-09-16T04:42:35.960 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013628410000000001 2024-09-16T04:42:35.960 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.605718+0000", 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22917667899999999 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.605799+0000", 2024-09-16T04:42:35.961 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0592000000000004e-05 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=71, last_complete=79'55, committed=1, applied=1))", 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.866045+0000", 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.0038166750000004, 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40485729799999998, 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.962 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866045+0000", 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866045+0000", 2024-09-16T04:42:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866046+0000", 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.37e-07 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866052+0000", 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3520000000000003e-06 2024-09-16T04:42:35.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866061+0000", 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5300000000000002e-06 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866067+0000", 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5740000000000003e-06 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.270839+0000", 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40477243800000001 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.270902+0000", 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2767000000000004e-05 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.966 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=72, last_complete=79'57, committed=1, applied=1))", 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.866722+0000", 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.0031392260000001, 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40500170400000002, 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.967 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866722+0000", 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866722+0000", 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866723+0000", 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7400000000000002e-07 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866728+0000", 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4229999999999998e-06 2024-09-16T04:42:35.969 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866738+0000", 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0876000000000001e-05 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866745+0000", 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6669999999999999e-06 2024-09-16T04:42:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271420+0000", 2024-09-16T04:42:35.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40467444499999999 2024-09-16T04:42:35.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271724+0000", 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000304519 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.972 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=72, last_complete=79'57, committed=1, applied=1))", 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.0028198909999997, 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40407422999999998, 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867046+0000", 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8739999999999997e-06 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867060+0000", 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4056999999999999e-05 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867067+0000", 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7560000000000002e-06 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271073+0000", 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40400594099999998 2024-09-16T04:42:35.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271116+0000", 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2970999999999999e-05 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=73, last_complete=79'58, committed=1, applied=1))", 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.970076+0000", 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.8997858589999996, 2024-09-16T04:42:35.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40209012, 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970076+0000", 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970076+0000", 2024-09-16T04:42:35.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970078+0000", 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.827e-06 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970088+0000", 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0769e-05 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970114+0000", 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5839000000000001e-05 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970127+0000", 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3118999999999999e-05 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372126+0000", 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40199895699999999 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372166+0000", 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9609e-05 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=73, last_complete=79'58, committed=1, applied=1))", 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.970504+0000", 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.8993576460000003, 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40299647599999999, 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:35.980 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970504+0000", 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970504+0000", 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970505+0000", 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.173e-06 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970516+0000", 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1270000000000001e-05 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970538+0000", 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1622000000000001e-05 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.070638+0000", 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100099904 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372240+0000", 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30160231700000001 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373500+0000", 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00126019 2024-09-16T04:42:35.983 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.984 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.984 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.984 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:35.984 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:35.984 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:302: done (3 left) 2024-09-16T04:42:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:303: done (2 left) 2024-09-16T04:42:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:304: done (1 left) 2024-09-16T04:42:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:305: done (0 left) 2024-09-16T04:42:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:306: delete oid 44 current snap is 24 2024-09-16T04:42:35.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:42:35.987 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:35.983+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:36.086 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:36.083+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:36.118 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:36.119 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:36.119 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:36.119 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:36.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:42:36.188 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:36.183+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:36.201 INFO:tasks.rados.rados.0.smithi165.stdout:306: done (0 left) 2024-09-16T04:42:36.201 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 25 current snap is 24 2024-09-16T04:42:36.201 INFO:tasks.rados.rados.0.smithi165.stdout:307: seq_num 117 ranges {1769472=425984,2195456=606208,2801664=540672,3342336=409600} 2024-09-16T04:42:36.237 INFO:tasks.rados.rados.0.smithi165.stdout:307: writing smithi16530038-25 from 1769472 to 2195456 tid 1 2024-09-16T04:42:36.248 INFO:tasks.rados.rados.0.smithi165.stdout:307: writing smithi16530038-25 from 2195456 to 2801664 tid 2 2024-09-16T04:42:36.256 INFO:tasks.rados.rados.0.smithi165.stdout:307: writing smithi16530038-25 from 2801664 to 3342336 tid 3 2024-09-16T04:42:36.260 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:36.260 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:36.260 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:36.260 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:36.260 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:36.261 INFO:tasks.rados.rados.0.smithi165.stdout:307: writing smithi16530038-25 from 3342336 to 3751936 tid 4 2024-09-16T04:42:36.261 INFO:tasks.rados.rados.0.smithi165.stdout:308: delete oid 42 current snap is 24 2024-09-16T04:42:36.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:42:36.288 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:36.283+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:36.389 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:36.383+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:36.441 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T04:42:36.442 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:42:36.444 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=29, reqid=client.4728.0:365, at_version=49'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 37.171115532000002, 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15627164299999999, 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.445 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271409+0000", 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:42:36.446 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271765+0000", 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00035584100000000002 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271795+0000", 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9595999999999999e-05 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271820+0000", 2024-09-16T04:42:36.447 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5301999999999999e-05 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425575+0000", 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15375470799999999 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425593+0000", 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.448 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427644+0000", 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020504630000000002 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427680+0000", 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6613999999999999e-05 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.449 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/32 ECSubWrite(tid=86, reqid=client.4728.0:424, at_version=53'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.103007851000001, 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10390139399999999, 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.450 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339517+0000", 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.167e-06 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.451 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339944+0000", 2024-09-16T04:42:36.452 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00042670199999999999 2024-09-16T04:42:36.452 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.452 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.452 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.452 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339971+0000", 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6916000000000001e-05 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339982+0000", 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1445e-05 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.440244+0000", 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100261356 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.454 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.440271+0000", 2024-09-16T04:42:36.454 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7209e-05 2024-09-16T04:42:36.454 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.454 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.454 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.454 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.443349+0000", 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030780949999999999 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.443418+0000", 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8503999999999993e-05 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.455 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=118, reqid=client.4728.0:458, at_version=58'27, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 28.037109520000001, 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20346340800000001, 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.456 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405416+0000", 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.221e-06 2024-09-16T04:42:36.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405551+0000", 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013492800000000001 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405567+0000", 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6552999999999999e-05 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505670+0000", 2024-09-16T04:42:36.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10010258399999999 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505840+0000", 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017047199999999999 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505892+0000", 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1406000000000002e-05 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608838+0000", 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10294645299999999 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608878+0000", 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9790999999999999e-05 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=119, reqid=client.4728.0:459, at_version=58'28, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.505750+0000", 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.936773703, 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10316181200000001, 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505750+0000", 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.461 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505750+0000", 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505752+0000", 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5069999999999999e-06 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505837+0000", 2024-09-16T04:42:36.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5020999999999998e-05 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505877+0000", 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0028999999999999e-05 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505901+0000", 2024-09-16T04:42:36.463 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3833e-05 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507589+0000", 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00168877 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507620+0000", 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0366e-05 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608884+0000", 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101264361 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608912+0000", 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7925000000000001e-05 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=120, reqid=client.4728.0:460, at_version=58'29, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.936578767, 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10344765, 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:36.466 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505946+0000", 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.467 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505979+0000", 2024-09-16T04:42:36.468 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3062000000000003e-05 2024-09-16T04:42:36.468 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.468 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.468 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.506027+0000", 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7333000000000002e-05 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.506050+0000", 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2918999999999999e-05 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608654+0000", 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102604337 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.469 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608663+0000", 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.986e-06 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.609362+0000", 2024-09-16T04:42:36.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00069878900000000001 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.609393+0000", 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1241000000000001e-05 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 59/32 ECSubWrite(tid=119, reqid=osd.11.0:119, at_version=59'48, trim_to=0'0, roll_forward_to=58'44))", 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:09.238987+0000", 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.203537154999999, 2024-09-16T04:42:36.471 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102666459, 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.238987+0000", 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.238987+0000", 2024-09-16T04:42:36.472 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.238993+0000", 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0090000000000004e-06 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.239025+0000", 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2252999999999998e-05 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.239046+0000", 2024-09-16T04:42:36.473 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0738999999999999e-05 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.339146+0000", 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100099935 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.339239+0000", 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3381000000000006e-05 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.474 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.339249+0000", 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.028e-05 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.341604+0000", 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023544400000000002 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.341653+0000", 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9422000000000002e-05 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.475 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/32 ECSubWrite(tid=132, reqid=osd.4.0:132, at_version=60'35, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.979655379, 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20257109500000001, 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:36.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3300000000000003e-07 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462925+0000", 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6362e-05 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462958+0000", 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2876000000000002e-05 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462991+0000", 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2836000000000002e-05 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.478 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463182+0000", 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000190736 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463210+0000", 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7925000000000001e-05 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665388+0000", 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202177734 2024-09-16T04:42:36.479 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665440+0000", 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2193e-05 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.480 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.978697140000001, 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20164931799999999, 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.481 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463828+0000", 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.161e-06 2024-09-16T04:42:36.482 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463862+0000", 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3846000000000001e-05 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463881+0000", 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9596e-05 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463895+0000", 2024-09-16T04:42:36.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3448999999999999e-05 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.464025+0000", 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000129677 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.464038+0000", 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3654e-05 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.484 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665451+0000", 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201412806 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665476+0000", 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5128999999999999e-05 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 65/32 ECSubWrite(tid=59, reqid=osd.2.0:59, at_version=65'40, trim_to=0'0, roll_forward_to=64'36))", 2024-09-16T04:42:36.485 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.990544087, 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202052495, 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451982+0000", 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.227e-06 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452027+0000", 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5142000000000002e-05 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.487 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452061+0000", 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3529999999999999e-05 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452079+0000", 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7941999999999999e-05 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.488 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.652452+0000", 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200372941 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.652468+0000", 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6517000000000001e-05 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.653953+0000", 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001484857 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.654032+0000", 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9338999999999994e-05 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=75, reqid=client.4728.0:577, at_version=69'50, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.113506348000001, 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20012123100000001, 2024-09-16T04:42:36.490 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.491 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329019+0000", 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.654e-06 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329563+0000", 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054386599999999995 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.492 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329591+0000", 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7759999999999998e-05 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329619+0000", 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7968999999999999e-05 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427652+0000", 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098032790999999994 2024-09-16T04:42:36.493 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427662+0000", 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0736e-05 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529104+0000", 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10144188 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529139+0000", 2024-09-16T04:42:36.494 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4575e-05 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=76, reqid=client.4728.0:578, at_version=69'51, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.104399584999999, 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19168868999999999, 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338125+0000", 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9099999999999991e-07 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338313+0000", 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018766200000000001 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338329+0000", 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5807e-05 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.497 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338339+0000", 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0451e-05 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.528499+0000", 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19015979599999999 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.528508+0000", 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0469999999999999e-06 2024-09-16T04:42:36.498 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529785+0000", 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001276758 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529813+0000", 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8178e-05 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.499 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=77, reqid=client.4728.0:579, at_version=69'52, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.100567013999999, 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18830155900000001, 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.500 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341958+0000", 2024-09-16T04:42:36.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0400000000000005e-07 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342250+0000", 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029209100000000002 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342265+0000", 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4808e-05 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.502 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342273+0000", 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3760000000000008e-06 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529207+0000", 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18693383999999999 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529216+0000", 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9109999999999999e-06 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.503 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.530224+0000", 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00100815 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.530259+0000", 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4579000000000002e-05 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 69/67 ECSubWrite(tid=148, reqid=client.4728.0:575, at_version=69'47, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:36.504 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.032268338000002, 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102672033, 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:36.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410256+0000", 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410258+0000", 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1299999999999999e-06 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410345+0000", 2024-09-16T04:42:36.506 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6930999999999998e-05 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410382+0000", 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7292999999999997e-05 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.410407+0000", 2024-09-16T04:42:36.507 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5029999999999999e-05 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511669+0000", 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101262251 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511687+0000", 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7390000000000001e-05 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.512888+0000", 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012010910000000001 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.512928+0000", 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9916999999999997e-05 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.509 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 70/67 ECSubWrite(tid=152, reqid=osd.4.0:152, at_version=70'50, trim_to=0'0, roll_forward_to=69'48))", 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 16.121887318999999, 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.14250185500000001, 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.510 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5300000000000004e-07 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320696+0000", 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8539000000000003e-05 2024-09-16T04:42:36.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320732+0000", 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6368999999999997e-05 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320759+0000", 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6466999999999999e-05 2024-09-16T04:42:36.513 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.514 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.514 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320930+0000", 2024-09-16T04:42:36.514 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017177699999999999 2024-09-16T04:42:36.514 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320966+0000", 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5382000000000001e-05 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463102+0000", 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.14213593499999999 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463139+0000", 2024-09-16T04:42:36.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6733000000000002e-05 2024-09-16T04:42:36.516 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.516 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.516 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 70/67 ECSubWrite(tid=153, reqid=osd.4.0:153, at_version=70'52, trim_to=0'0, roll_forward_to=69'48))", 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:20.321873+0000", 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 16.120651418000001, 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.14129620000000001, 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.517 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321873+0000", 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321873+0000", 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321875+0000", 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.03e-06 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321927+0000", 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2876e-05 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.519 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321958+0000", 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0960000000000002e-05 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321981+0000", 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2742e-05 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.422274+0000", 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10029260199999999 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.422289+0000", 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5319999999999999e-05 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463146+0000", 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.040857346000000003 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463169+0000", 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2323999999999998e-05 2024-09-16T04:42:36.521 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.522 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.522 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.522 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 70/67 ECSubWrite(tid=150, reqid=osd.11.0:150, at_version=70'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:20.573428+0000", 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.869096101, 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103122031, 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.523 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.573428+0000", 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.573428+0000", 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.573429+0000", 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.142e-06 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.573466+0000", 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6894999999999998e-05 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.573490+0000", 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4392e-05 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.673601+0000", 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100110514 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.673775+0000", 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000174057 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.673801+0000", 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6063000000000001e-05 2024-09-16T04:42:36.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.676477+0000", 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026761530000000001 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.676550+0000", 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2815e-05 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 71/67 ECSubWrite(tid=156, reqid=osd.4.0:156, at_version=71'54, trim_to=0'0, roll_forward_to=70'52))", 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.283443274, 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201114086, 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0400000000000002e-07 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159105+0000", 2024-09-16T04:42:36.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3816999999999999e-05 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159199+0000", 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3851999999999995e-05 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.259300+0000", 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100100827 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.359431+0000", 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131028 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.359440+0000", 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3300000000000005e-06 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.360143+0000", 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00070308700000000003 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.360195+0000", 2024-09-16T04:42:36.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1841000000000001e-05 2024-09-16T04:42:36.532 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.532 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.532 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 74/67 ECSubWrite(tid=97, reqid=client.4728.0:613, at_version=74'68, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.370209+0000", 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.072314909999999, 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20209957100000001, 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370209+0000", 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370209+0000", 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370210+0000", 2024-09-16T04:42:36.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2300000000000002e-07 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370720+0000", 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000510762 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370739+0000", 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8434e-05 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.470817+0000", 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10007798699999999 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.470878+0000", 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.109e-05 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.470888+0000", 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0392e-05 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.572280+0000", 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10139131899999999 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.572309+0000", 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9164000000000001e-05 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.bs3,RELEASE,e:75,from:4(0),reservation_nonce:0)", 2024-09-16T04:42:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:25.281000+0000", 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.16152364, 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.198626372, 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281000+0000", 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281000+0000", 2024-09-16T04:42:36.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281001+0000", 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3499999999999995e-07 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281005+0000", 2024-09-16T04:42:36.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.245e-06 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281019+0000", 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3633e-05 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281027+0000", 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7309999999999993e-06 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.479556+0000", 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19852925199999999 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.479595+0000", 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9357000000000003e-05 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.479627+0000", 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1418999999999999e-05 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.541 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.1bs4,QREQUEST,e:75,from:10(0),reservation_nonce:5)", 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:25.349066+0000", 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.093458467, 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13250140399999999, 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:36.542 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349066+0000", 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349066+0000", 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:36.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349068+0000", 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7130000000000001e-06 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349082+0000", 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3593e-05 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349120+0000", 2024-09-16T04:42:36.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7858999999999998e-05 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349137+0000", 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7303e-05 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.481493+0000", 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13235649399999999 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.481518+0000", 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4783999999999999e-05 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.481567+0000", 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8658000000000001e-05 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.546 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.547 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:36.547 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:36.547 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:36.548 INFO:tasks.rados.rados.0.smithi165.stdout:307: finishing write tid 1 to smithi16530038-25 2024-09-16T04:42:36.549 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:42:36.550 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T04:42:36.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:36.552 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:36.543+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:36.559 INFO:tasks.rados.rados.0.smithi165.stdout:307: finishing write tid 2 to smithi16530038-25 2024-09-16T04:42:36.649 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:36.643+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:36.682 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:36.682 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:36.682 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:36.682 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:36.697 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:42:36.751 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:36.743+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:36.821 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:36.821 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:36.821 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:36.821 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:36.821 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:36.836 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:42:36.852 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:36.847+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:36.856 INFO:tasks.rados.rados.0.smithi165.stdout:307: finishing write tid 3 to smithi16530038-25 2024-09-16T04:42:36.857 INFO:tasks.rados.rados.0.smithi165.stdout:307: finishing write tid 4 to smithi16530038-25 2024-09-16T04:42:36.918 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:36.919 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:42:36.938 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-16T04:42:36.443069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":86237747,"num_objects":104,"num_object_clones":53,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":6,"num_read":283,"num_read_kb":69468,"num_write":487,"num_write_kb":44206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7979008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1541,"ondisk_log_size":1541,"up":270,"acting":270,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":277,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":657704,"kb_used_data":162832,"kb_used_omap":24,"kb_used_meta":494503,"kb_avail":1509291736,"statfs":{"total":1546188226560,"available":1545514737664,"internally_reserved":0,"allocated":166739968,"data_stored":224516734,"data_compressed":2977743,"data_compressed_allocated":4096000,"data_compressed_original":9052160,"omap_allocated":25522,"internal_metadata":506371150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,4,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4882432,"num_objects":1,"num_object_clones":2,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":-1,"num_read":-8,"num_read_kb":-2041,"num_write":17,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.035846"},"pg_stats":[{"pgid":"3.23","version":"60'17","reported_seq":94,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.414138+0000","last_change":"2024-09-16T04:42:32.527925+0000","last_active":"2024-09-16T04:42:33.414138+0000","last_peered":"2024-09-16T04:42:33.414138+0000","last_clean":"2024-09-16T04:42:33.414138+0000","last_became_active":"2024-09-16T04:42:20.647183+0000","last_became_peered":"2024-09-16T04:42:20.647183+0000","last_unstale":"2024-09-16T04:42:33.414138+0000","last_undegraded":"2024-09-16T04:42:33.414138+0000","last_fullsized":"2024-09-16T04:42:33.414138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'17","last_scrub_stamp":"2024-09-16T04:42:25.580862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.580862+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:26.007009+0000","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0.100913348,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,0,15,12,1],"acting":[8,4,11,0,15,12,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.22","version":"72'17","reported_seq":118,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.651439+0000","last_change":"2024-09-16T04:42:33.023687+0000","last_active":"2024-09-16T04:42:34.651439+0000","last_peered":"2024-09-16T04:42:34.651439+0000","last_clean":"2024-09-16T04:42:34.651439+0000","last_became_active":"2024-09-16T04:42:20.464635+0000","last_became_peered":"2024-09-16T04:42:20.464635+0000","last_unstale":"2024-09-16T04:42:34.651439+0000","last_undegraded":"2024-09-16T04:42:34.651439+0000","last_fullsized":"2024-09-16T04:42:34.651439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":6,"last_scrub":"72'17","last_scrub_stamp":"2024-09-16T04:42:26.786330+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.786330+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:48.271094+0000","scrub_duration":843,"objects_trimmed":0,"snaptrim_duration":0.39897545000000001,"stat_sum":{"num_bytes":2048000,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,11,4,7,14],"acting":[2,5,0,11,4,7,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.21","version":"82'37","reported_seq":123,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.651339+0000","last_change":"2024-09-16T04:42:32.921864+0000","last_active":"2024-09-16T04:42:34.651339+0000","last_peered":"2024-09-16T04:42:34.651339+0000","last_clean":"2024-09-16T04:42:34.651339+0000","last_became_active":"2024-09-16T04:42:20.469617+0000","last_became_peered":"2024-09-16T04:42:20.469617+0000","last_unstale":"2024-09-16T04:42:34.651339+0000","last_undegraded":"2024-09-16T04:42:34.651339+0000","last_fullsized":"2024-09-16T04:42:34.651339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'31","last_scrub_stamp":"2024-09-16T04:42:26.688435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.688435+0000","objects_scrubbed":1,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:54.387887+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0.397550448,"stat_sum":{"num_bytes":311296,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,5,15,4,0,12],"acting":[2,10,5,15,4,0,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.20","version":"36'4","reported_seq":94,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.722758+0000","last_change":"2024-09-16T04:42:32.429202+0000","last_active":"2024-09-16T04:42:33.722758+0000","last_peered":"2024-09-16T04:42:33.722758+0000","last_clean":"2024-09-16T04:42:33.722758+0000","last_became_active":"2024-09-16T04:42:20.360418+0000","last_became_peered":"2024-09-16T04:42:20.360418+0000","last_unstale":"2024-09-16T04:42:33.722758+0000","last_undegraded":"2024-09-16T04:42:33.722758+0000","last_fullsized":"2024-09-16T04:42:33.722758+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":6,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.091321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.091321+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:29.355039+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0.0023327389999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,13,5,3,14,15],"acting":[6,2,13,5,3,14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1f","version":"79'56","reported_seq":123,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.555671+0000","last_change":"2024-09-16T04:42:32.427427+0000","last_active":"2024-09-16T04:42:35.555671+0000","last_peered":"2024-09-16T04:42:35.555671+0000","last_clean":"2024-09-16T04:42:35.555671+0000","last_became_active":"2024-09-16T04:42:20.460902+0000","last_became_peered":"2024-09-16T04:42:20.460902+0000","last_unstale":"2024-09-16T04:42:35.555671+0000","last_undegraded":"2024-09-16T04:42:35.555671+0000","last_fullsized":"2024-09-16T04:42:35.555671+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'52","last_scrub_stamp":"2024-09-16T04:42:25.895250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.895250+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.653646+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0.002688204,"stat_sum":{"num_bytes":1867776,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,15,8,0,2,11,14],"acting":[1,15,8,0,2,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1e","version":"82'67","reported_seq":161,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.413765+0000","last_change":"2024-09-16T04:42:32.828165+0000","last_active":"2024-09-16T04:42:33.413765+0000","last_peered":"2024-09-16T04:42:33.413765+0000","last_clean":"2024-09-16T04:42:33.413765+0000","last_became_active":"2024-09-16T04:42:20.562288+0000","last_became_peered":"2024-09-16T04:42:20.562288+0000","last_unstale":"2024-09-16T04:42:33.413765+0000","last_undegraded":"2024-09-16T04:42:33.413765+0000","last_fullsized":"2024-09-16T04:42:33.413765+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"74'53","last_scrub_stamp":"2024-09-16T04:42:27.214170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.214170+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:51.442021+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0.20073812399999999,"stat_sum":{"num_bytes":3047424,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":8930,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,5,1,14,3,10],"acting":[8,6,5,1,14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1d","version":"83'133","reported_seq":201,"reported_epoch":84,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:34.767327+0000","last_change":"2024-09-16T04:42:34.359042+0000","last_active":"2024-09-16T04:42:34.767327+0000","last_peered":"2024-09-16T04:42:34.767327+0000","last_clean":"2024-09-16T04:42:34.767327+0000","last_became_active":"2024-09-16T04:42:26.481390+0000","last_became_peered":"2024-09-16T04:42:26.481390+0000","last_unstale":"2024-09-16T04:42:34.767327+0000","last_undegraded":"2024-09-16T04:42:34.767327+0000","last_fullsized":"2024-09-16T04:42:34.767327+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.31264773099999998,"stat_sum":{"num_bytes":12009472,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":13428,"num_write":38,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,11,9,14,13,3],"acting":[5,15,11,9,14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1c","version":"82'56","reported_seq":338,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:42:32.719365+0000","last_change":"2024-09-16T04:42:32.720204+0000","last_active":"2024-09-16T04:42:32.719365+0000","last_peered":"2024-09-16T04:42:32.719365+0000","last_clean":"2024-09-16T04:42:32.719365+0000","last_became_active":"2024-09-16T04:42:20.367551+0000","last_became_peered":"2024-09-16T04:42:20.367551+0000","last_unstale":"2024-09-16T04:42:32.719365+0000","last_undegraded":"2024-09-16T04:42:32.719365+0000","last_fullsized":"2024-09-16T04:42:32.719365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'54","last_scrub_stamp":"2024-09-16T04:42:25.992723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.992723+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:42.121377+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.30621248400000001,"stat_sum":{"num_bytes":1736704,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,4,13,5,12,9],"acting":[11,7,4,13,5,12,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1b","version":"63'44","reported_seq":94,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.535206+0000","last_change":"2024-09-16T04:42:32.521532+0000","last_active":"2024-09-16T04:42:33.535206+0000","last_peered":"2024-09-16T04:42:33.535206+0000","last_clean":"2024-09-16T04:42:33.535206+0000","last_became_active":"2024-09-16T04:42:20.564769+0000","last_became_peered":"2024-09-16T04:42:20.564769+0000","last_unstale":"2024-09-16T04:42:33.535206+0000","last_undegraded":"2024-09-16T04:42:33.535206+0000","last_fullsized":"2024-09-16T04:42:33.535206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'44","last_scrub_stamp":"2024-09-16T04:42:25.681915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.681915+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:46.598086+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0.102580914,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,5,8,12,13,7],"acting":[10,15,5,8,12,13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1a","version":"82'51","reported_seq":135,"reported_epoch":84,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:34.753286+0000","last_change":"2024-09-16T04:42:34.263891+0000","last_active":"2024-09-16T04:42:34.753286+0000","last_peered":"2024-09-16T04:42:34.753286+0000","last_clean":"2024-09-16T04:42:34.753286+0000","last_became_active":"2024-09-16T04:42:20.461907+0000","last_became_peered":"2024-09-16T04:42:20.461907+0000","last_unstale":"2024-09-16T04:42:34.753286+0000","last_undegraded":"2024-09-16T04:42:34.753286+0000","last_fullsized":"2024-09-16T04:42:34.753286+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21466125899999999,"stat_sum":{"num_bytes":2080768,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":1638,"num_write":44,"num_write_kb":3120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12,10,13,9,7,11],"acting":[14,12,10,13,9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.19","version":"72'36","reported_seq":289,"reported_epoch":84,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T04:42:34.437693+0000","last_change":"2024-09-16T04:42:09.757868+0000","last_active":"2024-09-16T04:42:34.437693+0000","last_peered":"2024-09-16T04:42:34.437693+0000","last_clean":"2024-09-16T04:42:06.828625+0000","last_became_active":"2024-09-16T04:42:09.651531+0000","last_became_peered":"2024-09-16T04:42:09.651531+0000","last_unstale":"2024-09-16T04:42:34.437693+0000","last_undegraded":"2024-09-16T04:42:34.437693+0000","last_fullsized":"2024-09-16T04:42:34.437693+0000","mapping_epoch":58,"log_start":"44'22","ondisk_log_start":"44'22","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":22,"ondisk_log_size":14,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.098972921000000005,"stat_sum":{"num_bytes":2637824,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":10899,"num_write":5,"num_write_kb":672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[14,1,0,3,15,8,10],"avail_no_missing":["14(0)","0(2)","1(1)","3(3)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"0(2),1(1),3(3),8(5),10(6),14(0),15(4)","objects":6}],"blocked_by":[],"up_primary":8,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.18","version":"44'8","reported_seq":320,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.449376+0000","last_change":"2024-09-16T04:42:32.422664+0000","last_active":"2024-09-16T04:42:35.449376+0000","last_peered":"2024-09-16T04:42:35.449376+0000","last_clean":"2024-09-16T04:42:35.449376+0000","last_became_active":"2024-09-16T04:42:12.629523+0000","last_became_peered":"2024-09-16T04:42:12.629523+0000","last_unstale":"2024-09-16T04:42:35.449376+0000","last_undegraded":"2024-09-16T04:42:35.449376+0000","last_fullsized":"2024-09-16T04:42:35.449376+0000","mapping_epoch":61,"log_start":"44'8","ondisk_log_start":"44'8","created":55,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'8","last_scrub_stamp":"2024-09-16T04:42:14.378029+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:14.378029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.370201+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0.0051254639999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,0,4,5,7,12],"acting":[13,6,0,4,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.17","version":"44'6","reported_seq":191,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.466259+0000","last_change":"2024-09-16T04:42:32.412241+0000","last_active":"2024-09-16T04:42:34.466259+0000","last_peered":"2024-09-16T04:42:34.466259+0000","last_clean":"2024-09-16T04:42:34.466259+0000","last_became_active":"2024-09-16T04:42:04.604172+0000","last_became_peered":"2024-09-16T04:42:04.604172+0000","last_unstale":"2024-09-16T04:42:34.466259+0000","last_undegraded":"2024-09-16T04:42:34.466259+0000","last_fullsized":"2024-09-16T04:42:34.466259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'6","last_scrub_stamp":"2024-09-16T04:42:11.540212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:11.540212+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:29.243911+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0.001967828,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1777,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.16","version":"82'12","reported_seq":175,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:42:32.725662+0000","last_change":"2024-09-16T04:42:32.726357+0000","last_active":"2024-09-16T04:42:32.725662+0000","last_peered":"2024-09-16T04:42:32.725662+0000","last_clean":"2024-09-16T04:42:32.725662+0000","last_became_active":"2024-09-16T04:42:04.405792+0000","last_became_peered":"2024-09-16T04:42:04.405792+0000","last_unstale":"2024-09-16T04:42:32.725662+0000","last_undegraded":"2024-09-16T04:42:32.725662+0000","last_fullsized":"2024-09-16T04:42:32.725662+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:57.977558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:57.977558+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:04.280794+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.31192534500000002,"stat_sum":{"num_bytes":81920,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.15","version":"36'6","reported_seq":200,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.454997+0000","last_change":"2024-09-16T04:42:32.621397+0000","last_active":"2024-09-16T04:42:35.454997+0000","last_peered":"2024-09-16T04:42:35.454997+0000","last_clean":"2024-09-16T04:42:35.454997+0000","last_became_active":"2024-09-16T04:42:05.419352+0000","last_became_peered":"2024-09-16T04:42:05.419352+0000","last_unstale":"2024-09-16T04:42:35.454997+0000","last_undegraded":"2024-09-16T04:42:35.454997+0000","last_fullsized":"2024-09-16T04:42:35.454997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'6","last_scrub_stamp":"2024-09-16T04:42:10.772708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.772708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:15.799929+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.102408735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.14","version":"82'75","reported_seq":252,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.772273+0000","last_change":"2024-09-16T04:42:33.035351+0000","last_active":"2024-09-16T04:42:34.772273+0000","last_peered":"2024-09-16T04:42:34.772273+0000","last_clean":"2024-09-16T04:42:34.772273+0000","last_became_active":"2024-09-16T04:42:04.626975+0000","last_became_peered":"2024-09-16T04:42:04.626975+0000","last_unstale":"2024-09-16T04:42:34.772273+0000","last_undegraded":"2024-09-16T04:42:34.772273+0000","last_fullsized":"2024-09-16T04:42:34.772273+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:10.486582+0000","scrub_duration":899,"objects_trimmed":2,"snaptrim_duration":0.61650163800000002,"stat_sum":{"num_bytes":8290304,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":5331,"num_write":15,"num_write_kb":5072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4423680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.531006+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:42:33.531006+0000","last_peered":"2024-09-16T04:42:33.531006+0000","last_clean":"2024-09-16T04:42:33.531006+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:42:33.531006+0000","last_undegraded":"2024-09-16T04:42:33.531006+0000","last_fullsized":"2024-09-16T04:42:33.531006+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":247,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:32.519027+0000","last_change":"2024-09-16T04:42:32.519077+0000","last_active":"2024-09-16T04:42:32.519027+0000","last_peered":"2024-09-16T04:42:32.519027+0000","last_clean":"2024-09-16T04:42:32.519027+0000","last_became_active":"2024-09-16T04:41:48.483004+0000","last_became_peered":"2024-09-16T04:41:48.483004+0000","last_unstale":"2024-09-16T04:42:32.519027+0000","last_undegraded":"2024-09-16T04:42:32.519027+0000","last_fullsized":"2024-09-16T04:42:32.519027+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:07.452027+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.10123945099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":129,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.413939+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:42:33.413939+0000","last_peered":"2024-09-16T04:42:33.413939+0000","last_clean":"2024-09-16T04:42:33.413939+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:42:33.413939+0000","last_undegraded":"2024-09-16T04:42:33.413939+0000","last_fullsized":"2024-09-16T04:42:33.413939+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":257,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.839187+0000","last_change":"2024-09-16T04:42:32.421084+0000","last_active":"2024-09-16T04:42:34.839187+0000","last_peered":"2024-09-16T04:42:34.839187+0000","last_clean":"2024-09-16T04:42:34.839187+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:42:34.839187+0000","last_undegraded":"2024-09-16T04:42:34.839187+0000","last_fullsized":"2024-09-16T04:42:34.839187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:42:00.916431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.916431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:22.260170+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.001034348,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.722990+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:42:33.722990+0000","last_peered":"2024-09-16T04:42:33.722990+0000","last_clean":"2024-09-16T04:42:33.722990+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:42:33.722990+0000","last_undegraded":"2024-09-16T04:42:33.722990+0000","last_fullsized":"2024-09-16T04:42:33.722990+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"36'10","reported_seq":273,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.515547+0000","last_change":"2024-09-16T04:42:32.528429+0000","last_active":"2024-09-16T04:42:33.515547+0000","last_peered":"2024-09-16T04:42:33.515547+0000","last_clean":"2024-09-16T04:42:33.515547+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:42:33.515547+0000","last_undegraded":"2024-09-16T04:42:33.515547+0000","last_fullsized":"2024-09-16T04:42:33.515547+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:59.997918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.997918+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:00.901119+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.001092897,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.4","version":"0'0","reported_seq":129,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.738977+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:42:34.738977+0000","last_peered":"2024-09-16T04:42:34.738977+0000","last_clean":"2024-09-16T04:42:34.738977+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:42:34.738977+0000","last_undegraded":"2024-09-16T04:42:34.738977+0000","last_fullsized":"2024-09-16T04:42:34.738977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"82'60","reported_seq":315,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.530669+0000","last_change":"2024-09-16T04:42:33.429037+0000","last_active":"2024-09-16T04:42:33.530669+0000","last_peered":"2024-09-16T04:42:33.530669+0000","last_clean":"2024-09-16T04:42:33.530669+0000","last_became_active":"2024-09-16T04:41:48.475648+0000","last_became_peered":"2024-09-16T04:41:48.475648+0000","last_unstale":"2024-09-16T04:42:33.530669+0000","last_undegraded":"2024-09-16T04:42:33.530669+0000","last_fullsized":"2024-09-16T04:42:33.530669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:01.748213+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.81117342100000001,"stat_sum":{"num_bytes":5652480,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":8,"num_read_kb":675,"num_write":10,"num_write_kb":2192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.3","version":"82'29","reported_seq":304,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.449925+0000","last_change":"2024-09-16T04:42:32.821614+0000","last_active":"2024-09-16T04:42:35.449925+0000","last_peered":"2024-09-16T04:42:35.449925+0000","last_clean":"2024-09-16T04:42:35.449925+0000","last_became_active":"2024-09-16T04:42:18.590615+0000","last_became_peered":"2024-09-16T04:42:18.590615+0000","last_unstale":"2024-09-16T04:42:35.449925+0000","last_undegraded":"2024-09-16T04:42:35.449925+0000","last_fullsized":"2024-09-16T04:42:35.449925+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'21","last_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_deep_scrub":"72'21","last_deep_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.317975+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.609331+0000","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0.40184421199999998,"stat_sum":{"num_bytes":4358144,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.2","version":"28'2","reported_seq":133,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.795008+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:42:34.795008+0000","last_peered":"2024-09-16T04:42:34.795008+0000","last_clean":"2024-09-16T04:42:34.795008+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:42:34.795008+0000","last_undegraded":"2024-09-16T04:42:34.795008+0000","last_fullsized":"2024-09-16T04:42:34.795008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":276,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.436753+0000","last_change":"2024-09-16T04:42:32.414075+0000","last_active":"2024-09-16T04:42:34.436753+0000","last_peered":"2024-09-16T04:42:34.436753+0000","last_clean":"2024-09-16T04:42:34.436753+0000","last_became_active":"2024-09-16T04:42:18.592746+0000","last_became_peered":"2024-09-16T04:42:18.592746+0000","last_unstale":"2024-09-16T04:42:34.436753+0000","last_undegraded":"2024-09-16T04:42:34.436753+0000","last_fullsized":"2024-09-16T04:42:34.436753+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.089771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.089771+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:38.509443+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.0039071940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.549524+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:42:35.549524+0000","last_peered":"2024-09-16T04:42:35.549524+0000","last_clean":"2024-09-16T04:42:35.549524+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:42:35.549524+0000","last_undegraded":"2024-09-16T04:42:35.549524+0000","last_fullsized":"2024-09-16T04:42:35.549524+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"82'47","reported_seq":333,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.738828+0000","last_change":"2024-09-16T04:42:32.830329+0000","last_active":"2024-09-16T04:42:34.738828+0000","last_peered":"2024-09-16T04:42:34.738828+0000","last_clean":"2024-09-16T04:42:34.738828+0000","last_became_active":"2024-09-16T04:42:18.991363+0000","last_became_peered":"2024-09-16T04:42:18.991363+0000","last_unstale":"2024-09-16T04:42:34.738828+0000","last_undegraded":"2024-09-16T04:42:34.738828+0000","last_fullsized":"2024-09-16T04:42:34.738828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'35","last_scrub_stamp":"2024-09-16T04:42:24.144068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.144068+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:49.019212+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0.31067514299999999,"stat_sum":{"num_bytes":2637824,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":131,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.771914+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:42:34.771914+0000","last_peered":"2024-09-16T04:42:34.771914+0000","last_clean":"2024-09-16T04:42:34.771914+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:42:34.771914+0000","last_undegraded":"2024-09-16T04:42:34.771914+0000","last_fullsized":"2024-09-16T04:42:34.771914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"59'15","reported_seq":289,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.450023+0000","last_change":"2024-09-16T04:42:32.420184+0000","last_active":"2024-09-16T04:42:35.450023+0000","last_peered":"2024-09-16T04:42:35.450023+0000","last_clean":"2024-09-16T04:42:35.450023+0000","last_became_active":"2024-09-16T04:42:18.699582+0000","last_became_peered":"2024-09-16T04:42:18.699582+0000","last_unstale":"2024-09-16T04:42:35.450023+0000","last_undegraded":"2024-09-16T04:42:35.450023+0000","last_fullsized":"2024-09-16T04:42:35.450023+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'15","last_scrub_stamp":"2024-09-16T04:42:26.595325+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.595325+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.018553+0000","scrub_duration":967,"objects_trimmed":0,"snaptrim_duration":0.0026559349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.3","version":"26'1","reported_seq":132,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.465737+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:42:34.465737+0000","last_peered":"2024-09-16T04:42:34.465737+0000","last_clean":"2024-09-16T04:42:34.465737+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:42:34.465737+0000","last_undegraded":"2024-09-16T04:42:34.465737+0000","last_fullsized":"2024-09-16T04:42:34.465737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":330,"reported_epoch":85,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:35.858384+0000","last_change":"2024-09-16T04:42:35.756670+0000","last_active":"2024-09-16T04:42:35.858384+0000","last_peered":"2024-09-16T04:42:35.858384+0000","last_clean":"2024-09-16T04:42:35.858384+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:42:35.858384+0000","last_undegraded":"2024-09-16T04:42:35.858384+0000","last_fullsized":"2024-09-16T04:42:35.858384+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"82'27","reported_seq":317,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.465862+0000","last_change":"2024-09-16T04:42:32.927820+0000","last_active":"2024-09-16T04:42:34.465862+0000","last_peered":"2024-09-16T04:42:34.465862+0000","last_clean":"2024-09-16T04:42:34.465862+0000","last_became_active":"2024-09-16T04:42:06.535331+0000","last_became_peered":"2024-09-16T04:42:06.535331+0000","last_unstale":"2024-09-16T04:42:34.465862+0000","last_undegraded":"2024-09-16T04:42:34.465862+0000","last_fullsized":"2024-09-16T04:42:34.465862+0000","mapping_epoch":32,"log_start":"44'8","ondisk_log_start":"44'8","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-09-16T04:42:13.576020+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:13.576020+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":12,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:18.472629+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0.51746854600000003,"stat_sum":{"num_bytes":2752512,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.9","version":"50'30","reported_seq":316,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.536495+0000","last_change":"2024-09-16T04:42:32.412446+0000","last_active":"2024-09-16T04:42:34.536495+0000","last_peered":"2024-09-16T04:42:34.536495+0000","last_clean":"2024-09-16T04:42:34.536495+0000","last_became_active":"2024-09-16T04:42:06.419535+0000","last_became_peered":"2024-09-16T04:42:06.419535+0000","last_unstale":"2024-09-16T04:42:34.536495+0000","last_undegraded":"2024-09-16T04:42:34.536495+0000","last_fullsized":"2024-09-16T04:42:34.536495+0000","mapping_epoch":32,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'30","last_scrub_stamp":"2024-09-16T04:42:10.554025+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.554025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:35.999815+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.00266329,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.a","version":"82'57","reported_seq":333,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.794517+0000","last_change":"2024-09-16T04:42:33.226910+0000","last_active":"2024-09-16T04:42:34.794517+0000","last_peered":"2024-09-16T04:42:34.794517+0000","last_clean":"2024-09-16T04:42:34.794517+0000","last_became_active":"2024-09-16T04:42:18.694273+0000","last_became_peered":"2024-09-16T04:42:18.694273+0000","last_unstale":"2024-09-16T04:42:34.794517+0000","last_undegraded":"2024-09-16T04:42:34.794517+0000","last_fullsized":"2024-09-16T04:42:34.794517+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'41","last_scrub_stamp":"2024-09-16T04:42:25.530017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.530017+0000","objects_scrubbed":5,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:26.092340+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.80727635799999997,"stat_sum":{"num_bytes":458752,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":4,"num_read_kb":241,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.b","version":"82'67","reported_seq":333,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.738994+0000","last_change":"2024-09-16T04:42:33.435767+0000","last_active":"2024-09-16T04:42:34.738994+0000","last_peered":"2024-09-16T04:42:34.738994+0000","last_clean":"2024-09-16T04:42:34.738994+0000","last_became_active":"2024-09-16T04:42:18.688040+0000","last_became_peered":"2024-09-16T04:42:18.688040+0000","last_unstale":"2024-09-16T04:42:34.738994+0000","last_undegraded":"2024-09-16T04:42:34.738994+0000","last_fullsized":"2024-09-16T04:42:34.738994+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'58","last_scrub_stamp":"2024-09-16T04:42:25.280206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.280206+0000","objects_scrubbed":6,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:35.557447+0000","scrub_duration":511,"objects_trimmed":1,"snaptrim_duration":0.91546229800000001,"stat_sum":{"num_bytes":7864320,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":8535,"num_write":4,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.c","version":"80'73","reported_seq":319,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:32.726407+0000","last_change":"2024-09-16T04:42:32.726450+0000","last_active":"2024-09-16T04:42:32.726407+0000","last_peered":"2024-09-16T04:42:32.726407+0000","last_clean":"2024-09-16T04:42:32.726407+0000","last_became_active":"2024-09-16T04:42:18.798872+0000","last_became_peered":"2024-09-16T04:42:18.798872+0000","last_unstale":"2024-09-16T04:42:32.726407+0000","last_undegraded":"2024-09-16T04:42:32.726407+0000","last_fullsized":"2024-09-16T04:42:32.726407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:37.580465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50895387299999995,"stat_sum":{"num_bytes":3506176,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":3127,"num_write":53,"num_write_kb":3904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.d","version":"82'156","reported_seq":418,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.676787+0000","last_change":"2024-09-16T04:42:33.331438+0000","last_active":"2024-09-16T04:42:34.676787+0000","last_peered":"2024-09-16T04:42:34.676787+0000","last_clean":"2024-09-16T04:42:34.676787+0000","last_became_active":"2024-09-16T04:42:19.196254+0000","last_became_peered":"2024-09-16T04:42:19.196254+0000","last_unstale":"2024-09-16T04:42:34.676787+0000","last_undegraded":"2024-09-16T04:42:34.676787+0000","last_fullsized":"2024-09-16T04:42:34.676787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'134","last_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_deep_scrub":"74'134","last_deep_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.664276+0000","objects_scrubbed":9,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:53.345945+0000","scrub_duration":1071,"objects_trimmed":1,"snaptrim_duration":0.90785241699999997,"stat_sum":{"num_bytes":6537216,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":4,"num_read_kb":4920,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.e","version":"82'79","reported_seq":343,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.651510+0000","last_change":"2024-09-16T04:42:34.647272+0000","last_active":"2024-09-16T04:42:34.651510+0000","last_peered":"2024-09-16T04:42:34.651510+0000","last_clean":"2024-09-16T04:42:34.651510+0000","last_became_active":"2024-09-16T04:42:18.700111+0000","last_became_peered":"2024-09-16T04:42:18.700111+0000","last_unstale":"2024-09-16T04:42:34.651510+0000","last_undegraded":"2024-09-16T04:42:34.651510+0000","last_fullsized":"2024-09-16T04:42:34.651510+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:34.646929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.646929+0000","objects_scrubbed":6,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:00.469327+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.80398320099999998,"stat_sum":{"num_bytes":5668864,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.f","version":"82'72","reported_seq":315,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:32.720239+0000","last_change":"2024-09-16T04:42:32.720274+0000","last_active":"2024-09-16T04:42:32.720239+0000","last_peered":"2024-09-16T04:42:32.720239+0000","last_clean":"2024-09-16T04:42:32.720239+0000","last_became_active":"2024-09-16T04:42:18.793319+0000","last_became_peered":"2024-09-16T04:42:18.793319+0000","last_unstale":"2024-09-16T04:42:32.720239+0000","last_undegraded":"2024-09-16T04:42:32.720239+0000","last_fullsized":"2024-09-16T04:42:32.720239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:37.580442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40908640699999999,"stat_sum":{"num_bytes":2793472,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":868,"num_write":33,"num_write_kb":3528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.10","version":"82'19","reported_seq":148,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.751636+0000","last_change":"2024-09-16T04:42:32.833414+0000","last_active":"2024-09-16T04:42:34.751636+0000","last_peered":"2024-09-16T04:42:34.751636+0000","last_clean":"2024-09-16T04:42:34.751636+0000","last_became_active":"2024-09-16T04:42:16.560449+0000","last_became_peered":"2024-09-16T04:42:16.560449+0000","last_unstale":"2024-09-16T04:42:34.751636+0000","last_undegraded":"2024-09-16T04:42:34.751636+0000","last_fullsized":"2024-09-16T04:42:34.751636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:42:24.882865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.882865+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.563758+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0.30920652599999998,"stat_sum":{"num_bytes":3162112,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1,14,5,3,15],"acting":[2,7,1,14,5,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.11","version":"82'38","reported_seq":226,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.535263+0000","last_change":"2024-09-16T04:42:32.829747+0000","last_active":"2024-09-16T04:42:33.535263+0000","last_peered":"2024-09-16T04:42:33.535263+0000","last_clean":"2024-09-16T04:42:33.535263+0000","last_became_active":"2024-09-16T04:42:04.623974+0000","last_became_peered":"2024-09-16T04:42:04.623974+0000","last_unstale":"2024-09-16T04:42:33.535263+0000","last_undegraded":"2024-09-16T04:42:33.535263+0000","last_fullsized":"2024-09-16T04:42:33.535263+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'28","last_scrub_stamp":"2024-09-16T04:42:10.158979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.158979+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:24.493965+0000","scrub_duration":855,"objects_trimmed":0,"snaptrim_duration":0.310515557,"stat_sum":{"num_bytes":3653632,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":5,"num_read_kb":5233,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.12","version":"82'26","reported_seq":206,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.454661+0000","last_change":"2024-09-16T04:42:32.924963+0000","last_active":"2024-09-16T04:42:35.454661+0000","last_peered":"2024-09-16T04:42:35.454661+0000","last_clean":"2024-09-16T04:42:35.454661+0000","last_became_active":"2024-09-16T04:42:04.406048+0000","last_became_peered":"2024-09-16T04:42:04.406048+0000","last_unstale":"2024-09-16T04:42:35.454661+0000","last_undegraded":"2024-09-16T04:42:35.454661+0000","last_fullsized":"2024-09-16T04:42:35.454661+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'12","last_scrub_stamp":"2024-09-16T04:42:10.115913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.115913+0000","objects_scrubbed":1,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:32.579364+0000","scrub_duration":611,"objects_trimmed":0,"snaptrim_duration":0.40624049299999998,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3745,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.13","version":"36'3","reported_seq":162,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:42:32.415482+0000","last_change":"2024-09-16T04:42:32.416226+0000","last_active":"2024-09-16T04:42:32.415482+0000","last_peered":"2024-09-16T04:42:32.415482+0000","last_clean":"2024-09-16T04:42:32.415482+0000","last_became_active":"2024-09-16T04:42:04.415210+0000","last_became_peered":"2024-09-16T04:42:04.415210+0000","last_unstale":"2024-09-16T04:42:32.415482+0000","last_undegraded":"2024-09-16T04:42:32.415482+0000","last_fullsized":"2024-09-16T04:42:32.415482+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:42:10.690368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.690368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:21.698249+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.0022148710000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,0,4,10],"acting":[11,7,9,13,0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":85647360,"num_objects":100,"num_object_clones":53,"num_object_copies":700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":6,"num_read":147,"num_read_kb":69350,"num_write":282,"num_write_kb":40744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7979008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":151638016,"data_stored":211378176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1381,"ondisk_log_size":1381,"up":252,"acting":252,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247827,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42008,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329832,"statfs":{"total":96636764160,"available":96593747968,"internally_reserved":0,"allocated":11079680,"data_stored":16614579,"data_compressed":219308,"data_compressed_allocated":299008,"data_compressed_original":1028096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247827,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42624,"kb_used_data":11432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329216,"statfs":{"total":96636764160,"available":96593117184,"internally_reserved":0,"allocated":11706368,"data_stored":15747560,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,6,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247827,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42272,"kb_used_data":11084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329568,"statfs":{"total":96636764160,"available":96593477632,"internally_reserved":0,"allocated":11350016,"data_stored":15059432,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313233,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42184,"kb_used_data":10992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329656,"statfs":{"total":96636764160,"available":96593567744,"internally_reserved":0,"allocated":11255808,"data_stored":16156768,"data_compressed":178366,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313233,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40908,"kb_used_data":9724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330932,"statfs":{"total":96636764160,"available":96594874368,"internally_reserved":0,"allocated":9957376,"data_stored":12257376,"data_compressed":178366,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280530,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39256,"kb_used_data":8068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332584,"statfs":{"total":96636764160,"available":96596566016,"internally_reserved":0,"allocated":8261632,"data_stored":10277725,"data_compressed":213311,"data_compressed_allocated":290816,"data_compressed_original":1011712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280529,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42640,"kb_used_data":11452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329200,"statfs":{"total":96636764160,"available":96593100800,"internally_reserved":0,"allocated":11726848,"data_stored":14967101,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":11032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":11296768,"data_stored":15129083,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280530,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41928,"kb_used_data":10736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329912,"statfs":{"total":96636764160,"available":96593829888,"internally_reserved":0,"allocated":10993664,"data_stored":15696189,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280530,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41412,"kb_used_data":10224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330428,"statfs":{"total":96636764160,"available":96594358272,"internally_reserved":0,"allocated":10469376,"data_stored":12603709,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40428,"kb_used_data":9240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331412,"statfs":{"total":96636764160,"available":96595365888,"internally_reserved":0,"allocated":9461760,"data_stored":12742973,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":9636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":9867264,"data_stored":13564392,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43260,"kb_used_data":12072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328580,"statfs":{"total":96636764160,"available":96592465920,"internally_reserved":0,"allocated":12361728,"data_stored":17207613,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280530,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40084,"kb_used_data":8896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331756,"statfs":{"total":96636764160,"available":96595718144,"internally_reserved":0,"allocated":9109504,"data_stored":13246781,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":6048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":6193152,"data_stored":7778640,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42568,"kb_used_data":11376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329272,"statfs":{"total":96636764160,"available":96593174528,"internally_reserved":0,"allocated":11649024,"data_stored":15466813,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10317824,"data_stored":15478784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10387456,"data_stored":14360576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7249920,"data_stored":9060352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10330112,"data_stored":14430208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10043392,"data_stored":15007744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9519104,"data_stored":11915264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":12054528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8904704,"data_stored":12865536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11411456,"data_stored":16519168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10743808,"data_stored":15048704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8159232,"data_stored":12558336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8884224,"data_stored":11444224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":5238784,"data_stored":7090176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":10043392,"data_stored":14069760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10698752,"data_stored":14778368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":11194368,"data_stored":14696448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:36.941 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:36.953 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:36.947+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=10, reqid=client.4728.0:122, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 51.315419691999999, 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201524707, 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684051+0000", 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.277e-06 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684081+0000", 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0414999999999999e-05 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684109+0000", 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7455000000000001e-05 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684125+0000", 2024-09-16T04:42:37.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5707000000000001e-05 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884504+0000", 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200379487 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884519+0000", 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4953000000000001e-05 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885504+0000", 2024-09-16T04:42:37.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098537600000000005 2024-09-16T04:42:37.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885574+0000", 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0036999999999994e-05 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.009 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 51.025098092999997, 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20260964000000001, 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:37.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974373+0000", 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.683e-06 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974841+0000", 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468255 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974891+0000", 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0198000000000003e-05 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.074965+0000", 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100073226 2024-09-16T04:42:37.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175170+0000", 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100205119 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175184+0000", 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4626e-05 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176898+0000", 2024-09-16T04:42:37.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017136390000000001 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176981+0000", 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2893999999999996e-05 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:160, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 50.545806483, 2024-09-16T04:42:37.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20191688999999999, 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453664+0000", 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453695+0000", 2024-09-16T04:42:37.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0998999999999999e-05 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453714+0000", 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9745000000000002e-05 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.553888+0000", 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10017351100000001 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654137+0000", 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100248949 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654151+0000", 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4279e-05 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655526+0000", 2024-09-16T04:42:37.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013750520000000001 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655580+0000", 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3634999999999997e-05 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.as2 43/32 ECSubRead(tid=37, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 43.791054760000002, 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200544685, 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208415+0000", 2024-09-16T04:42:37.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.34e-07 2024-09-16T04:42:37.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208425+0000", 2024-09-16T04:42:37.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8859999999999999e-06 2024-09-16T04:42:37.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.208457+0000", 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1644999999999999e-05 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.308556+0000", 2024-09-16T04:42:37.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100099178 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.408782+0000", 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100226105 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:53.408959+0000", 2024-09-16T04:42:37.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017743699999999999 2024-09-16T04:42:37.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.028 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.029 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 48/32 ECSubWrite(tid=44, reqid=client.4728.0:362, at_version=48'17, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.642783182999999, 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205456467, 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356687+0000", 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356985+0000", 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029765100000000001 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357014+0000", 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9853e-05 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357034+0000", 2024-09-16T04:42:37.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9592000000000001e-05 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557308+0000", 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20027399500000001 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557322+0000", 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4346e-05 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562072+0000", 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047490960000000004 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562143+0000", 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1191000000000003e-05 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s6 57/53 ECSubWrite(tid=9, reqid=osd.10.0:9, at_version=57'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:37.034 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 28.770352276000001, 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209615832, 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229119+0000", 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229208+0000", 2024-09-16T04:42:37.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9190999999999994e-05 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229249+0000", 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0842e-05 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329348+0000", 2024-09-16T04:42:37.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10009905600000001 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426747+0000", 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.09739892 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426767+0000", 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0173000000000002e-05 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438659+0000", 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011892171 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438733+0000", 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3515000000000003e-05 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.039 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.040 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.040 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s3 63/55 ECSubWrite(tid=22, reqid=osd.5.0:22, at_version=63'14, trim_to=0'0, roll_forward_to=62'12))", 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.421073135, 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20251555199999999, 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9999999999999999e-07 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578420+0000", 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3271000000000001e-05 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578449+0000", 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9488999999999998e-05 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.678595+0000", 2024-09-16T04:42:37.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100145814 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778824+0000", 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100228703 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778840+0000", 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6123000000000001e-05 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780838+0000", 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019980129999999999 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780912+0000", 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3938999999999998e-05 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es2 69/67 ECSubWrite(tid=80, reqid=client.4728.0:581, at_version=0'0, trim_to=0'0, roll_forward_to=69'54))", 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.272898422000001, 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20154364599999999, 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726573+0000", 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1399999999999998e-06 2024-09-16T04:42:37.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726628+0000", 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4376000000000003e-05 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726668+0000", 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0466000000000002e-05 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.826832+0000", 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100163697 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827016+0000", 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018461700000000001 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827048+0000", 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2186e-05 2024-09-16T04:42:37.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928006+0000", 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10095765900000001 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928115+0000", 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010850500000000001 2024-09-16T04:42:37.050 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1es3 74/69 ECSubWrite(tid=62, reqid=client.4728.0:620, at_version=74'53, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.015619259999999, 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20195012000000001, 2024-09-16T04:42:37.051 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983851+0000", 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.173e-06 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983905+0000", 2024-09-16T04:42:37.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4027000000000003e-05 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983927+0000", 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2005999999999999e-05 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084064+0000", 2024-09-16T04:42:37.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10013664899999999 2024-09-16T04:42:37.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084267+0000", 2024-09-16T04:42:37.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020252399999999999 2024-09-16T04:42:37.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.056 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.056 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084288+0000", 2024-09-16T04:42:37.056 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1943999999999999e-05 2024-09-16T04:42:37.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.185701+0000", 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101412144 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.185800+0000", 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9653000000000003e-05 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.057 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDScrubReserve(3.1es3,QREQUEST,e:75,from:8(0),reservation_nonce:2)", 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:25.948827+0000", 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.050642835, 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200576114, 2024-09-16T04:42:37.058 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.948827+0000", 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.948827+0000", 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.948828+0000", 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9130000000000001e-06 2024-09-16T04:42:37.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.948838+0000", 2024-09-16T04:42:37.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0899999999999994e-06 2024-09-16T04:42:37.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.948866+0000", 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8269999999999999e-05 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:26.049004+0000", 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100137695 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:26.149244+0000", 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100240017 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:26.149297+0000", 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2995999999999999e-05 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:26.149403+0000", 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000106133 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:697 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~704512] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.5085637419999998, 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.26883937299999999, 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:37.065 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 697 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490907+0000", 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8050000000000001e-06 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491858+0000", 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095084100000000001 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491869+0000", 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1184e-05 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491878+0000", 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3480000000000004e-06 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491963+0000", 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5368e-05 2024-09-16T04:42:37.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491974+0000", 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1161e-05 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492735+0000", 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00076025099999999998 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492939+0000", 2024-09-16T04:42:37.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020470800000000001 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.494899+0000", 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019597260000000002 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.496840+0000", 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019404660000000001 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759668+0000", 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.262828544 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759745+0000", 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6971000000000006e-05 2024-09-16T04:42:37.071 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:698 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~655360] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.5075687819999999, 2024-09-16T04:42:37.072 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.36985094099999999, 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 698 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491902+0000", 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9100000000000002e-07 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492632+0000", 2024-09-16T04:42:37.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00073060200000000001 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492643+0000", 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0910999999999999e-05 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592713+0000", 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100069806 2024-09-16T04:42:37.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593257+0000", 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00054396700000000002 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593384+0000", 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012739399999999999 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.594523+0000", 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011388450000000001 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759798+0000", 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.16527468300000001 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.861671+0000", 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101873753 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.861752+0000", 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0088999999999998e-05 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:699 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~507904] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:37.078 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.4067261259999997, 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.37090961900000002, 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 699 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592745+0000", 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7e-06 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593160+0000", 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041519899999999999 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593172+0000", 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2248e-05 2024-09-16T04:42:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593182+0000", 2024-09-16T04:42:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4210000000000001e-06 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.595327+0000", 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0021456460000000002 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.595409+0000", 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1316000000000005e-05 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.596547+0000", 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011381710000000001 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759799+0000", 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.163252014 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963555+0000", 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203755827 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963653+0000", 2024-09-16T04:42:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8077000000000003e-05 2024-09-16T04:42:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:700 3.1fs0 3:fb308b36:::smithi16530038-23:head [setxattr _header (57) in=7b] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:37.087 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:37.087 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.406267766, 2024-09-16T04:42:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.37068284299999998, 2024-09-16T04:42:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 700 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593207+0000", 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7740000000000004e-06 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593217+0000", 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9329999999999996e-06 2024-09-16T04:42:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593225+0000", 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4749999999999996e-06 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.597569+0000", 2024-09-16T04:42:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0043444690000000001 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.597710+0000", 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141125 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.598396+0000", 2024-09-16T04:42:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000685767 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759801+0000", 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.16140457599999999 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963785+0000", 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20398479 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963884+0000", 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9166999999999994e-05 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.096 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.096 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:701 3.1fs0 3:fb308b36:::smithi16530038-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:37.096 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:37.096 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.4062237179999997, 2024-09-16T04:42:37.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.67642940299999998, 2024-09-16T04:42:37.097 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.097 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.097 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:37.097 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:37.097 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:37.097 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:37.097 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 701 2024-09-16T04:42:37.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.098 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:37.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.51e-07 2024-09-16T04:42:37.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593250+0000", 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0099999999999997e-06 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593260+0000", 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6360000000000006e-06 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593267+0000", 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2869999999999999e-06 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.600494+0000", 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072266040000000002 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.600669+0000", 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000174799 2024-09-16T04:42:37.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.964260+0000", 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.36359097400000001 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.964360+0000", 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000100375 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.269675+0000", 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.305315167 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s3 79/53 ECSubWrite(tid=126, reqid=client.4728.0:704, at_version=79'66, trim_to=0'0, roll_forward_to=72'63))", 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.2346901199999998, 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20206831, 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.103 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764781+0000", 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.796e-06 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764820+0000", 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8662999999999999e-05 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764850+0000", 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0349999999999999e-05 2024-09-16T04:42:37.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.864962+0000", 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100111618 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965141+0000", 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100179383 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965156+0000", 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4997000000000001e-05 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.966786+0000", 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016294300000000001 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.966848+0000", 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2073e-05 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1fs0 79/69 ECSubWriteReply(tid=4, last_complete=79'56, committed=1, applied=1))", 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.0338333520000003, 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202173098, 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965637+0000", 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9400000000000005e-07 2024-09-16T04:42:37.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965640+0000", 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4130000000000002e-06 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965650+0000", 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0207000000000001e-05 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965659+0000", 2024-09-16T04:42:37.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6759999999999996e-06 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167760+0000", 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202100745 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167809+0000", 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9462999999999997e-05 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1fs0 79/69 ECSubWriteReply(tid=4, last_complete=79'56, committed=1, applied=1))", 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.033694573, 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202105168, 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.112 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965777+0000", 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6700000000000001e-06 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965789+0000", 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2173e-05 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965816+0000", 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7270000000000001e-05 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965837+0000", 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.107e-05 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167854+0000", 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20201680599999999 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167880+0000", 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6179000000000001e-05 2024-09-16T04:42:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1fs0 79/69 ECSubRead(tid=5, to_read={3:fb308b36:::smithi16530038-23:head=0,4096,0}, subchunks={3:fb308b36:::smithi16530038-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.0322694730000004, 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20164594899999999, 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967275+0000", 2024-09-16T04:42:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4894999999999994e-05 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.067391+0000", 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100115909 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.168332+0000", 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100941066 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.168846+0000", 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051407899999999997 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 80/67 ECSubWrite(tid=131, reqid=osd.3.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=80'51))", 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:42:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.3023058499999998, 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30121982800000002, 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:42:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697165+0000", 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.623e-06 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697185+0000", 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0103e-05 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697215+0000", 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9733000000000001e-05 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697231+0000", 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5699999999999999e-05 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.897603+0000", 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20037271700000001 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.897620+0000", 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6983999999999999e-05 2024-09-16T04:42:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.998295+0000", 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100674176 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.998383+0000", 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8792e-05 2024-09-16T04:42:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:37.126 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:37.126 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:37.128 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:37.123+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:37.129 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:42:37.158 INFO:tasks.rados.rados.0.smithi165.stdout:307: finishing write tid 5 to smithi16530038-25 2024-09-16T04:42:37.160 INFO:tasks.rados.rados.0.smithi165.stdout:308: done (1 left) 2024-09-16T04:42:37.160 INFO:tasks.rados.rados.0.smithi165.stdout:309: snap_remove snap 20 2024-09-16T04:42:37.228 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:37.223+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:37.258 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:37.259 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:42:37.272 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:37.272 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:37.277 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-16T04:42:37.014047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":86237747,"num_objects":104,"num_object_clones":53,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":6,"num_read":283,"num_read_kb":69468,"num_write":487,"num_write_kb":44206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7979008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1541,"ondisk_log_size":1541,"up":270,"acting":270,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":277,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":657704,"kb_used_data":162832,"kb_used_omap":24,"kb_used_meta":494503,"kb_avail":1509291736,"statfs":{"total":1546188226560,"available":1545514737664,"internally_reserved":0,"allocated":166739968,"data_stored":224516734,"data_compressed":2977743,"data_compressed_allocated":4096000,"data_compressed_original":9052160,"omap_allocated":25522,"internal_metadata":506371150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,4,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4882432,"num_objects":1,"num_object_clones":2,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":-1,"num_read":-8,"num_read_kb":-2041,"num_write":17,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001839"},"pg_stats":[{"pgid":"3.23","version":"60'17","reported_seq":94,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.414138+0000","last_change":"2024-09-16T04:42:32.527925+0000","last_active":"2024-09-16T04:42:33.414138+0000","last_peered":"2024-09-16T04:42:33.414138+0000","last_clean":"2024-09-16T04:42:33.414138+0000","last_became_active":"2024-09-16T04:42:20.647183+0000","last_became_peered":"2024-09-16T04:42:20.647183+0000","last_unstale":"2024-09-16T04:42:33.414138+0000","last_undegraded":"2024-09-16T04:42:33.414138+0000","last_fullsized":"2024-09-16T04:42:33.414138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'17","last_scrub_stamp":"2024-09-16T04:42:25.580862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.580862+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:26.007009+0000","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0.100913348,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,0,15,12,1],"acting":[8,4,11,0,15,12,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.22","version":"72'17","reported_seq":118,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.651439+0000","last_change":"2024-09-16T04:42:33.023687+0000","last_active":"2024-09-16T04:42:34.651439+0000","last_peered":"2024-09-16T04:42:34.651439+0000","last_clean":"2024-09-16T04:42:34.651439+0000","last_became_active":"2024-09-16T04:42:20.464635+0000","last_became_peered":"2024-09-16T04:42:20.464635+0000","last_unstale":"2024-09-16T04:42:34.651439+0000","last_undegraded":"2024-09-16T04:42:34.651439+0000","last_fullsized":"2024-09-16T04:42:34.651439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":6,"last_scrub":"72'17","last_scrub_stamp":"2024-09-16T04:42:26.786330+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.786330+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:48.271094+0000","scrub_duration":843,"objects_trimmed":0,"snaptrim_duration":0.39897545000000001,"stat_sum":{"num_bytes":2048000,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,11,4,7,14],"acting":[2,5,0,11,4,7,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.21","version":"82'37","reported_seq":123,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.651339+0000","last_change":"2024-09-16T04:42:32.921864+0000","last_active":"2024-09-16T04:42:34.651339+0000","last_peered":"2024-09-16T04:42:34.651339+0000","last_clean":"2024-09-16T04:42:34.651339+0000","last_became_active":"2024-09-16T04:42:20.469617+0000","last_became_peered":"2024-09-16T04:42:20.469617+0000","last_unstale":"2024-09-16T04:42:34.651339+0000","last_undegraded":"2024-09-16T04:42:34.651339+0000","last_fullsized":"2024-09-16T04:42:34.651339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'31","last_scrub_stamp":"2024-09-16T04:42:26.688435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.688435+0000","objects_scrubbed":1,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:54.387887+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0.397550448,"stat_sum":{"num_bytes":311296,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,5,15,4,0,12],"acting":[2,10,5,15,4,0,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.20","version":"36'4","reported_seq":94,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.722758+0000","last_change":"2024-09-16T04:42:32.429202+0000","last_active":"2024-09-16T04:42:33.722758+0000","last_peered":"2024-09-16T04:42:33.722758+0000","last_clean":"2024-09-16T04:42:33.722758+0000","last_became_active":"2024-09-16T04:42:20.360418+0000","last_became_peered":"2024-09-16T04:42:20.360418+0000","last_unstale":"2024-09-16T04:42:33.722758+0000","last_undegraded":"2024-09-16T04:42:33.722758+0000","last_fullsized":"2024-09-16T04:42:33.722758+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":6,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.091321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.091321+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:29.355039+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0.0023327389999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,13,5,3,14,15],"acting":[6,2,13,5,3,14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1f","version":"79'56","reported_seq":123,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.555671+0000","last_change":"2024-09-16T04:42:32.427427+0000","last_active":"2024-09-16T04:42:35.555671+0000","last_peered":"2024-09-16T04:42:35.555671+0000","last_clean":"2024-09-16T04:42:35.555671+0000","last_became_active":"2024-09-16T04:42:20.460902+0000","last_became_peered":"2024-09-16T04:42:20.460902+0000","last_unstale":"2024-09-16T04:42:35.555671+0000","last_undegraded":"2024-09-16T04:42:35.555671+0000","last_fullsized":"2024-09-16T04:42:35.555671+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'52","last_scrub_stamp":"2024-09-16T04:42:25.895250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.895250+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.653646+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0.002688204,"stat_sum":{"num_bytes":1867776,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,15,8,0,2,11,14],"acting":[1,15,8,0,2,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1e","version":"82'67","reported_seq":161,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.413765+0000","last_change":"2024-09-16T04:42:32.828165+0000","last_active":"2024-09-16T04:42:33.413765+0000","last_peered":"2024-09-16T04:42:33.413765+0000","last_clean":"2024-09-16T04:42:33.413765+0000","last_became_active":"2024-09-16T04:42:20.562288+0000","last_became_peered":"2024-09-16T04:42:20.562288+0000","last_unstale":"2024-09-16T04:42:33.413765+0000","last_undegraded":"2024-09-16T04:42:33.413765+0000","last_fullsized":"2024-09-16T04:42:33.413765+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"74'53","last_scrub_stamp":"2024-09-16T04:42:27.214170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.214170+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:51.442021+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0.20073812399999999,"stat_sum":{"num_bytes":3047424,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":8930,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,5,1,14,3,10],"acting":[8,6,5,1,14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1d","version":"83'133","reported_seq":201,"reported_epoch":84,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:34.767327+0000","last_change":"2024-09-16T04:42:34.359042+0000","last_active":"2024-09-16T04:42:34.767327+0000","last_peered":"2024-09-16T04:42:34.767327+0000","last_clean":"2024-09-16T04:42:34.767327+0000","last_became_active":"2024-09-16T04:42:26.481390+0000","last_became_peered":"2024-09-16T04:42:26.481390+0000","last_unstale":"2024-09-16T04:42:34.767327+0000","last_undegraded":"2024-09-16T04:42:34.767327+0000","last_fullsized":"2024-09-16T04:42:34.767327+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.31264773099999998,"stat_sum":{"num_bytes":12009472,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":13428,"num_write":38,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,11,9,14,13,3],"acting":[5,15,11,9,14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1c","version":"82'56","reported_seq":338,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:42:32.719365+0000","last_change":"2024-09-16T04:42:32.720204+0000","last_active":"2024-09-16T04:42:32.719365+0000","last_peered":"2024-09-16T04:42:32.719365+0000","last_clean":"2024-09-16T04:42:32.719365+0000","last_became_active":"2024-09-16T04:42:20.367551+0000","last_became_peered":"2024-09-16T04:42:20.367551+0000","last_unstale":"2024-09-16T04:42:32.719365+0000","last_undegraded":"2024-09-16T04:42:32.719365+0000","last_fullsized":"2024-09-16T04:42:32.719365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'54","last_scrub_stamp":"2024-09-16T04:42:25.992723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.992723+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:42.121377+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.30621248400000001,"stat_sum":{"num_bytes":1736704,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,4,13,5,12,9],"acting":[11,7,4,13,5,12,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1b","version":"63'44","reported_seq":94,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.535206+0000","last_change":"2024-09-16T04:42:32.521532+0000","last_active":"2024-09-16T04:42:33.535206+0000","last_peered":"2024-09-16T04:42:33.535206+0000","last_clean":"2024-09-16T04:42:33.535206+0000","last_became_active":"2024-09-16T04:42:20.564769+0000","last_became_peered":"2024-09-16T04:42:20.564769+0000","last_unstale":"2024-09-16T04:42:33.535206+0000","last_undegraded":"2024-09-16T04:42:33.535206+0000","last_fullsized":"2024-09-16T04:42:33.535206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'44","last_scrub_stamp":"2024-09-16T04:42:25.681915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.681915+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:46.598086+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0.102580914,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,5,8,12,13,7],"acting":[10,15,5,8,12,13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.1a","version":"82'51","reported_seq":135,"reported_epoch":84,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:34.753286+0000","last_change":"2024-09-16T04:42:34.263891+0000","last_active":"2024-09-16T04:42:34.753286+0000","last_peered":"2024-09-16T04:42:34.753286+0000","last_clean":"2024-09-16T04:42:34.753286+0000","last_became_active":"2024-09-16T04:42:20.461907+0000","last_became_peered":"2024-09-16T04:42:20.461907+0000","last_unstale":"2024-09-16T04:42:34.753286+0000","last_undegraded":"2024-09-16T04:42:34.753286+0000","last_fullsized":"2024-09-16T04:42:34.753286+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21466125899999999,"stat_sum":{"num_bytes":2080768,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":1638,"num_write":44,"num_write_kb":3120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12,10,13,9,7,11],"acting":[14,12,10,13,9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.19","version":"72'36","reported_seq":289,"reported_epoch":84,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T04:42:34.437693+0000","last_change":"2024-09-16T04:42:09.757868+0000","last_active":"2024-09-16T04:42:34.437693+0000","last_peered":"2024-09-16T04:42:34.437693+0000","last_clean":"2024-09-16T04:42:06.828625+0000","last_became_active":"2024-09-16T04:42:09.651531+0000","last_became_peered":"2024-09-16T04:42:09.651531+0000","last_unstale":"2024-09-16T04:42:34.437693+0000","last_undegraded":"2024-09-16T04:42:34.437693+0000","last_fullsized":"2024-09-16T04:42:34.437693+0000","mapping_epoch":58,"log_start":"44'22","ondisk_log_start":"44'22","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":22,"ondisk_log_size":14,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.098972921000000005,"stat_sum":{"num_bytes":2637824,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":10899,"num_write":5,"num_write_kb":672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[14,1,0,3,15,8,10],"avail_no_missing":["14(0)","0(2)","1(1)","3(3)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"0(2),1(1),3(3),8(5),10(6),14(0),15(4)","objects":6}],"blocked_by":[],"up_primary":8,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.18","version":"44'8","reported_seq":320,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.449376+0000","last_change":"2024-09-16T04:42:32.422664+0000","last_active":"2024-09-16T04:42:35.449376+0000","last_peered":"2024-09-16T04:42:35.449376+0000","last_clean":"2024-09-16T04:42:35.449376+0000","last_became_active":"2024-09-16T04:42:12.629523+0000","last_became_peered":"2024-09-16T04:42:12.629523+0000","last_unstale":"2024-09-16T04:42:35.449376+0000","last_undegraded":"2024-09-16T04:42:35.449376+0000","last_fullsized":"2024-09-16T04:42:35.449376+0000","mapping_epoch":61,"log_start":"44'8","ondisk_log_start":"44'8","created":55,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'8","last_scrub_stamp":"2024-09-16T04:42:14.378029+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:14.378029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.370201+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0.0051254639999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,0,4,5,7,12],"acting":[13,6,0,4,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.17","version":"44'6","reported_seq":191,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.466259+0000","last_change":"2024-09-16T04:42:32.412241+0000","last_active":"2024-09-16T04:42:34.466259+0000","last_peered":"2024-09-16T04:42:34.466259+0000","last_clean":"2024-09-16T04:42:34.466259+0000","last_became_active":"2024-09-16T04:42:04.604172+0000","last_became_peered":"2024-09-16T04:42:04.604172+0000","last_unstale":"2024-09-16T04:42:34.466259+0000","last_undegraded":"2024-09-16T04:42:34.466259+0000","last_fullsized":"2024-09-16T04:42:34.466259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'6","last_scrub_stamp":"2024-09-16T04:42:11.540212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:11.540212+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:29.243911+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0.001967828,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1777,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.16","version":"82'12","reported_seq":175,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:42:32.725662+0000","last_change":"2024-09-16T04:42:32.726357+0000","last_active":"2024-09-16T04:42:32.725662+0000","last_peered":"2024-09-16T04:42:32.725662+0000","last_clean":"2024-09-16T04:42:32.725662+0000","last_became_active":"2024-09-16T04:42:04.405792+0000","last_became_peered":"2024-09-16T04:42:04.405792+0000","last_unstale":"2024-09-16T04:42:32.725662+0000","last_undegraded":"2024-09-16T04:42:32.725662+0000","last_fullsized":"2024-09-16T04:42:32.725662+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:57.977558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:57.977558+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:04.280794+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.31192534500000002,"stat_sum":{"num_bytes":81920,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.15","version":"36'6","reported_seq":200,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.454997+0000","last_change":"2024-09-16T04:42:32.621397+0000","last_active":"2024-09-16T04:42:35.454997+0000","last_peered":"2024-09-16T04:42:35.454997+0000","last_clean":"2024-09-16T04:42:35.454997+0000","last_became_active":"2024-09-16T04:42:05.419352+0000","last_became_peered":"2024-09-16T04:42:05.419352+0000","last_unstale":"2024-09-16T04:42:35.454997+0000","last_undegraded":"2024-09-16T04:42:35.454997+0000","last_fullsized":"2024-09-16T04:42:35.454997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'6","last_scrub_stamp":"2024-09-16T04:42:10.772708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.772708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:15.799929+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.102408735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.14","version":"82'75","reported_seq":252,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.772273+0000","last_change":"2024-09-16T04:42:33.035351+0000","last_active":"2024-09-16T04:42:34.772273+0000","last_peered":"2024-09-16T04:42:34.772273+0000","last_clean":"2024-09-16T04:42:34.772273+0000","last_became_active":"2024-09-16T04:42:04.626975+0000","last_became_peered":"2024-09-16T04:42:04.626975+0000","last_unstale":"2024-09-16T04:42:34.772273+0000","last_undegraded":"2024-09-16T04:42:34.772273+0000","last_fullsized":"2024-09-16T04:42:34.772273+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:10.486582+0000","scrub_duration":899,"objects_trimmed":2,"snaptrim_duration":0.61650163800000002,"stat_sum":{"num_bytes":8290304,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":5331,"num_write":15,"num_write_kb":5072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4423680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.531006+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:42:33.531006+0000","last_peered":"2024-09-16T04:42:33.531006+0000","last_clean":"2024-09-16T04:42:33.531006+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:42:33.531006+0000","last_undegraded":"2024-09-16T04:42:33.531006+0000","last_fullsized":"2024-09-16T04:42:33.531006+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":247,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:32.519027+0000","last_change":"2024-09-16T04:42:32.519077+0000","last_active":"2024-09-16T04:42:32.519027+0000","last_peered":"2024-09-16T04:42:32.519027+0000","last_clean":"2024-09-16T04:42:32.519027+0000","last_became_active":"2024-09-16T04:41:48.483004+0000","last_became_peered":"2024-09-16T04:41:48.483004+0000","last_unstale":"2024-09-16T04:42:32.519027+0000","last_undegraded":"2024-09-16T04:42:32.519027+0000","last_fullsized":"2024-09-16T04:42:32.519027+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:07.452027+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.10123945099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":129,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.413939+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:42:33.413939+0000","last_peered":"2024-09-16T04:42:33.413939+0000","last_clean":"2024-09-16T04:42:33.413939+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:42:33.413939+0000","last_undegraded":"2024-09-16T04:42:33.413939+0000","last_fullsized":"2024-09-16T04:42:33.413939+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":257,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.839187+0000","last_change":"2024-09-16T04:42:32.421084+0000","last_active":"2024-09-16T04:42:34.839187+0000","last_peered":"2024-09-16T04:42:34.839187+0000","last_clean":"2024-09-16T04:42:34.839187+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:42:34.839187+0000","last_undegraded":"2024-09-16T04:42:34.839187+0000","last_fullsized":"2024-09-16T04:42:34.839187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:42:00.916431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.916431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:22.260170+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.001034348,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.722990+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:42:33.722990+0000","last_peered":"2024-09-16T04:42:33.722990+0000","last_clean":"2024-09-16T04:42:33.722990+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:42:33.722990+0000","last_undegraded":"2024-09-16T04:42:33.722990+0000","last_fullsized":"2024-09-16T04:42:33.722990+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"36'10","reported_seq":273,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.515547+0000","last_change":"2024-09-16T04:42:32.528429+0000","last_active":"2024-09-16T04:42:33.515547+0000","last_peered":"2024-09-16T04:42:33.515547+0000","last_clean":"2024-09-16T04:42:33.515547+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:42:33.515547+0000","last_undegraded":"2024-09-16T04:42:33.515547+0000","last_fullsized":"2024-09-16T04:42:33.515547+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:59.997918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.997918+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:00.901119+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.001092897,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.4","version":"0'0","reported_seq":129,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.738977+0000","last_change":"2024-09-16T04:41:29.769865+0000","last_active":"2024-09-16T04:42:34.738977+0000","last_peered":"2024-09-16T04:42:34.738977+0000","last_clean":"2024-09-16T04:42:34.738977+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:42:34.738977+0000","last_undegraded":"2024-09-16T04:42:34.738977+0000","last_fullsized":"2024-09-16T04:42:34.738977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"82'60","reported_seq":315,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.530669+0000","last_change":"2024-09-16T04:42:33.429037+0000","last_active":"2024-09-16T04:42:33.530669+0000","last_peered":"2024-09-16T04:42:33.530669+0000","last_clean":"2024-09-16T04:42:33.530669+0000","last_became_active":"2024-09-16T04:41:48.475648+0000","last_became_peered":"2024-09-16T04:41:48.475648+0000","last_unstale":"2024-09-16T04:42:33.530669+0000","last_undegraded":"2024-09-16T04:42:33.530669+0000","last_fullsized":"2024-09-16T04:42:33.530669+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:01.748213+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.81117342100000001,"stat_sum":{"num_bytes":5652480,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":8,"num_read_kb":675,"num_write":10,"num_write_kb":2192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.3","version":"82'29","reported_seq":304,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.449925+0000","last_change":"2024-09-16T04:42:32.821614+0000","last_active":"2024-09-16T04:42:35.449925+0000","last_peered":"2024-09-16T04:42:35.449925+0000","last_clean":"2024-09-16T04:42:35.449925+0000","last_became_active":"2024-09-16T04:42:18.590615+0000","last_became_peered":"2024-09-16T04:42:18.590615+0000","last_unstale":"2024-09-16T04:42:35.449925+0000","last_undegraded":"2024-09-16T04:42:35.449925+0000","last_fullsized":"2024-09-16T04:42:35.449925+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'21","last_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_deep_scrub":"72'21","last_deep_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.317975+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.609331+0000","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0.40184421199999998,"stat_sum":{"num_bytes":4358144,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.2","version":"28'2","reported_seq":133,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.795008+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:42:34.795008+0000","last_peered":"2024-09-16T04:42:34.795008+0000","last_clean":"2024-09-16T04:42:34.795008+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:42:34.795008+0000","last_undegraded":"2024-09-16T04:42:34.795008+0000","last_fullsized":"2024-09-16T04:42:34.795008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":276,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.436753+0000","last_change":"2024-09-16T04:42:32.414075+0000","last_active":"2024-09-16T04:42:34.436753+0000","last_peered":"2024-09-16T04:42:34.436753+0000","last_clean":"2024-09-16T04:42:34.436753+0000","last_became_active":"2024-09-16T04:42:18.592746+0000","last_became_peered":"2024-09-16T04:42:18.592746+0000","last_unstale":"2024-09-16T04:42:34.436753+0000","last_undegraded":"2024-09-16T04:42:34.436753+0000","last_fullsized":"2024-09-16T04:42:34.436753+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.089771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.089771+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:38.509443+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.0039071940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.549524+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:42:35.549524+0000","last_peered":"2024-09-16T04:42:35.549524+0000","last_clean":"2024-09-16T04:42:35.549524+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:42:35.549524+0000","last_undegraded":"2024-09-16T04:42:35.549524+0000","last_fullsized":"2024-09-16T04:42:35.549524+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"82'47","reported_seq":333,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.738828+0000","last_change":"2024-09-16T04:42:32.830329+0000","last_active":"2024-09-16T04:42:34.738828+0000","last_peered":"2024-09-16T04:42:34.738828+0000","last_clean":"2024-09-16T04:42:34.738828+0000","last_became_active":"2024-09-16T04:42:18.991363+0000","last_became_peered":"2024-09-16T04:42:18.991363+0000","last_unstale":"2024-09-16T04:42:34.738828+0000","last_undegraded":"2024-09-16T04:42:34.738828+0000","last_fullsized":"2024-09-16T04:42:34.738828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'35","last_scrub_stamp":"2024-09-16T04:42:24.144068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.144068+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:49.019212+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0.31067514299999999,"stat_sum":{"num_bytes":2637824,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":131,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.771914+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:42:34.771914+0000","last_peered":"2024-09-16T04:42:34.771914+0000","last_clean":"2024-09-16T04:42:34.771914+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:42:34.771914+0000","last_undegraded":"2024-09-16T04:42:34.771914+0000","last_fullsized":"2024-09-16T04:42:34.771914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"59'15","reported_seq":289,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.450023+0000","last_change":"2024-09-16T04:42:32.420184+0000","last_active":"2024-09-16T04:42:35.450023+0000","last_peered":"2024-09-16T04:42:35.450023+0000","last_clean":"2024-09-16T04:42:35.450023+0000","last_became_active":"2024-09-16T04:42:18.699582+0000","last_became_peered":"2024-09-16T04:42:18.699582+0000","last_unstale":"2024-09-16T04:42:35.450023+0000","last_undegraded":"2024-09-16T04:42:35.450023+0000","last_fullsized":"2024-09-16T04:42:35.450023+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'15","last_scrub_stamp":"2024-09-16T04:42:26.595325+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.595325+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.018553+0000","scrub_duration":967,"objects_trimmed":0,"snaptrim_duration":0.0026559349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"2.3","version":"26'1","reported_seq":132,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.465737+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:42:34.465737+0000","last_peered":"2024-09-16T04:42:34.465737+0000","last_clean":"2024-09-16T04:42:34.465737+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:42:34.465737+0000","last_undegraded":"2024-09-16T04:42:34.465737+0000","last_fullsized":"2024-09-16T04:42:34.465737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":330,"reported_epoch":85,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:42:35.858384+0000","last_change":"2024-09-16T04:42:35.756670+0000","last_active":"2024-09-16T04:42:35.858384+0000","last_peered":"2024-09-16T04:42:35.858384+0000","last_clean":"2024-09-16T04:42:35.858384+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:42:35.858384+0000","last_undegraded":"2024-09-16T04:42:35.858384+0000","last_fullsized":"2024-09-16T04:42:35.858384+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:41:23.354934+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"82'27","reported_seq":317,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.465862+0000","last_change":"2024-09-16T04:42:32.927820+0000","last_active":"2024-09-16T04:42:34.465862+0000","last_peered":"2024-09-16T04:42:34.465862+0000","last_clean":"2024-09-16T04:42:34.465862+0000","last_became_active":"2024-09-16T04:42:06.535331+0000","last_became_peered":"2024-09-16T04:42:06.535331+0000","last_unstale":"2024-09-16T04:42:34.465862+0000","last_undegraded":"2024-09-16T04:42:34.465862+0000","last_fullsized":"2024-09-16T04:42:34.465862+0000","mapping_epoch":32,"log_start":"44'8","ondisk_log_start":"44'8","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-09-16T04:42:13.576020+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:13.576020+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":12,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:18.472629+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0.51746854600000003,"stat_sum":{"num_bytes":2752512,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.9","version":"50'30","reported_seq":316,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.536495+0000","last_change":"2024-09-16T04:42:32.412446+0000","last_active":"2024-09-16T04:42:34.536495+0000","last_peered":"2024-09-16T04:42:34.536495+0000","last_clean":"2024-09-16T04:42:34.536495+0000","last_became_active":"2024-09-16T04:42:06.419535+0000","last_became_peered":"2024-09-16T04:42:06.419535+0000","last_unstale":"2024-09-16T04:42:34.536495+0000","last_undegraded":"2024-09-16T04:42:34.536495+0000","last_fullsized":"2024-09-16T04:42:34.536495+0000","mapping_epoch":32,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'30","last_scrub_stamp":"2024-09-16T04:42:10.554025+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.554025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:35.999815+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.00266329,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.a","version":"82'57","reported_seq":333,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.794517+0000","last_change":"2024-09-16T04:42:33.226910+0000","last_active":"2024-09-16T04:42:34.794517+0000","last_peered":"2024-09-16T04:42:34.794517+0000","last_clean":"2024-09-16T04:42:34.794517+0000","last_became_active":"2024-09-16T04:42:18.694273+0000","last_became_peered":"2024-09-16T04:42:18.694273+0000","last_unstale":"2024-09-16T04:42:34.794517+0000","last_undegraded":"2024-09-16T04:42:34.794517+0000","last_fullsized":"2024-09-16T04:42:34.794517+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'41","last_scrub_stamp":"2024-09-16T04:42:25.530017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.530017+0000","objects_scrubbed":5,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:26.092340+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.80727635799999997,"stat_sum":{"num_bytes":458752,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":4,"num_read_kb":241,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.b","version":"82'67","reported_seq":333,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.738994+0000","last_change":"2024-09-16T04:42:33.435767+0000","last_active":"2024-09-16T04:42:34.738994+0000","last_peered":"2024-09-16T04:42:34.738994+0000","last_clean":"2024-09-16T04:42:34.738994+0000","last_became_active":"2024-09-16T04:42:18.688040+0000","last_became_peered":"2024-09-16T04:42:18.688040+0000","last_unstale":"2024-09-16T04:42:34.738994+0000","last_undegraded":"2024-09-16T04:42:34.738994+0000","last_fullsized":"2024-09-16T04:42:34.738994+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'58","last_scrub_stamp":"2024-09-16T04:42:25.280206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.280206+0000","objects_scrubbed":6,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:35.557447+0000","scrub_duration":511,"objects_trimmed":1,"snaptrim_duration":0.91546229800000001,"stat_sum":{"num_bytes":7864320,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":8535,"num_write":4,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.c","version":"80'73","reported_seq":319,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:32.726407+0000","last_change":"2024-09-16T04:42:32.726450+0000","last_active":"2024-09-16T04:42:32.726407+0000","last_peered":"2024-09-16T04:42:32.726407+0000","last_clean":"2024-09-16T04:42:32.726407+0000","last_became_active":"2024-09-16T04:42:18.798872+0000","last_became_peered":"2024-09-16T04:42:18.798872+0000","last_unstale":"2024-09-16T04:42:32.726407+0000","last_undegraded":"2024-09-16T04:42:32.726407+0000","last_fullsized":"2024-09-16T04:42:32.726407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:37.580465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50895387299999995,"stat_sum":{"num_bytes":3506176,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":3127,"num_write":53,"num_write_kb":3904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.d","version":"82'156","reported_seq":418,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.676787+0000","last_change":"2024-09-16T04:42:33.331438+0000","last_active":"2024-09-16T04:42:34.676787+0000","last_peered":"2024-09-16T04:42:34.676787+0000","last_clean":"2024-09-16T04:42:34.676787+0000","last_became_active":"2024-09-16T04:42:19.196254+0000","last_became_peered":"2024-09-16T04:42:19.196254+0000","last_unstale":"2024-09-16T04:42:34.676787+0000","last_undegraded":"2024-09-16T04:42:34.676787+0000","last_fullsized":"2024-09-16T04:42:34.676787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'134","last_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_deep_scrub":"74'134","last_deep_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.664276+0000","objects_scrubbed":9,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:53.345945+0000","scrub_duration":1071,"objects_trimmed":1,"snaptrim_duration":0.90785241699999997,"stat_sum":{"num_bytes":6537216,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":4,"num_read_kb":4920,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.e","version":"82'79","reported_seq":343,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.651510+0000","last_change":"2024-09-16T04:42:34.647272+0000","last_active":"2024-09-16T04:42:34.651510+0000","last_peered":"2024-09-16T04:42:34.651510+0000","last_clean":"2024-09-16T04:42:34.651510+0000","last_became_active":"2024-09-16T04:42:18.700111+0000","last_became_peered":"2024-09-16T04:42:18.700111+0000","last_unstale":"2024-09-16T04:42:34.651510+0000","last_undegraded":"2024-09-16T04:42:34.651510+0000","last_fullsized":"2024-09-16T04:42:34.651510+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:34.646929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.646929+0000","objects_scrubbed":6,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:00.469327+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.80398320099999998,"stat_sum":{"num_bytes":5668864,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.f","version":"82'72","reported_seq":315,"reported_epoch":82,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:32.720239+0000","last_change":"2024-09-16T04:42:32.720274+0000","last_active":"2024-09-16T04:42:32.720239+0000","last_peered":"2024-09-16T04:42:32.720239+0000","last_clean":"2024-09-16T04:42:32.720239+0000","last_became_active":"2024-09-16T04:42:18.793319+0000","last_became_peered":"2024-09-16T04:42:18.793319+0000","last_unstale":"2024-09-16T04:42:32.720239+0000","last_undegraded":"2024-09-16T04:42:32.720239+0000","last_fullsized":"2024-09-16T04:42:32.720239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:40.675693+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:37.580442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40908640699999999,"stat_sum":{"num_bytes":2793472,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":868,"num_write":33,"num_write_kb":3528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.10","version":"82'19","reported_seq":148,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T04:42:34.751636+0000","last_change":"2024-09-16T04:42:32.833414+0000","last_active":"2024-09-16T04:42:34.751636+0000","last_peered":"2024-09-16T04:42:34.751636+0000","last_clean":"2024-09-16T04:42:34.751636+0000","last_became_active":"2024-09-16T04:42:16.560449+0000","last_became_peered":"2024-09-16T04:42:16.560449+0000","last_unstale":"2024-09-16T04:42:34.751636+0000","last_undegraded":"2024-09-16T04:42:34.751636+0000","last_fullsized":"2024-09-16T04:42:34.751636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:42:24.882865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.882865+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.563758+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0.30920652599999998,"stat_sum":{"num_bytes":3162112,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1,14,5,3,15],"acting":[2,7,1,14,5,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.11","version":"82'38","reported_seq":226,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-16T04:42:33.535263+0000","last_change":"2024-09-16T04:42:32.829747+0000","last_active":"2024-09-16T04:42:33.535263+0000","last_peered":"2024-09-16T04:42:33.535263+0000","last_clean":"2024-09-16T04:42:33.535263+0000","last_became_active":"2024-09-16T04:42:04.623974+0000","last_became_peered":"2024-09-16T04:42:04.623974+0000","last_unstale":"2024-09-16T04:42:33.535263+0000","last_undegraded":"2024-09-16T04:42:33.535263+0000","last_fullsized":"2024-09-16T04:42:33.535263+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'28","last_scrub_stamp":"2024-09-16T04:42:10.158979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.158979+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:24.493965+0000","scrub_duration":855,"objects_trimmed":0,"snaptrim_duration":0.310515557,"stat_sum":{"num_bytes":3653632,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":5,"num_read_kb":5233,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.12","version":"82'26","reported_seq":206,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T04:42:35.454661+0000","last_change":"2024-09-16T04:42:32.924963+0000","last_active":"2024-09-16T04:42:35.454661+0000","last_peered":"2024-09-16T04:42:35.454661+0000","last_clean":"2024-09-16T04:42:35.454661+0000","last_became_active":"2024-09-16T04:42:04.406048+0000","last_became_peered":"2024-09-16T04:42:04.406048+0000","last_unstale":"2024-09-16T04:42:35.454661+0000","last_undegraded":"2024-09-16T04:42:35.454661+0000","last_fullsized":"2024-09-16T04:42:35.454661+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'12","last_scrub_stamp":"2024-09-16T04:42:10.115913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.115913+0000","objects_scrubbed":1,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:32.579364+0000","scrub_duration":611,"objects_trimmed":0,"snaptrim_duration":0.40624049299999998,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3745,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]},{"pgid":"3.13","version":"36'3","reported_seq":162,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-16T04:42:32.415482+0000","last_change":"2024-09-16T04:42:32.416226+0000","last_active":"2024-09-16T04:42:32.415482+0000","last_peered":"2024-09-16T04:42:32.415482+0000","last_clean":"2024-09-16T04:42:32.415482+0000","last_became_active":"2024-09-16T04:42:04.415210+0000","last_became_peered":"2024-09-16T04:42:04.415210+0000","last_unstale":"2024-09-16T04:42:32.415482+0000","last_undegraded":"2024-09-16T04:42:32.415482+0000","last_fullsized":"2024-09-16T04:42:32.415482+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:42:10.690368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.690368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:21.698249+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.0022148710000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,0,4,10],"acting":[11,7,9,13,0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":85647360,"num_objects":100,"num_object_clones":53,"num_object_copies":700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":6,"num_read":147,"num_read_kb":69350,"num_write":282,"num_write_kb":40744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7979008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":151638016,"data_stored":211378176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1381,"ondisk_log_size":1381,"up":252,"acting":252,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247827,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42008,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329832,"statfs":{"total":96636764160,"available":96593747968,"internally_reserved":0,"allocated":11079680,"data_stored":16614579,"data_compressed":219308,"data_compressed_allocated":299008,"data_compressed_original":1028096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247827,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42624,"kb_used_data":11432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329216,"statfs":{"total":96636764160,"available":96593117184,"internally_reserved":0,"allocated":11706368,"data_stored":15747560,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,6,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247827,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42272,"kb_used_data":11084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329568,"statfs":{"total":96636764160,"available":96593477632,"internally_reserved":0,"allocated":11350016,"data_stored":15059432,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313233,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42184,"kb_used_data":10992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329656,"statfs":{"total":96636764160,"available":96593567744,"internally_reserved":0,"allocated":11255808,"data_stored":16156768,"data_compressed":178366,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313233,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40908,"kb_used_data":9724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330932,"statfs":{"total":96636764160,"available":96594874368,"internally_reserved":0,"allocated":9957376,"data_stored":12257376,"data_compressed":178366,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280530,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39256,"kb_used_data":8068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332584,"statfs":{"total":96636764160,"available":96596566016,"internally_reserved":0,"allocated":8261632,"data_stored":10277725,"data_compressed":213311,"data_compressed_allocated":290816,"data_compressed_original":1011712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280529,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42640,"kb_used_data":11452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329200,"statfs":{"total":96636764160,"available":96593100800,"internally_reserved":0,"allocated":11726848,"data_stored":14967101,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":11032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":11296768,"data_stored":15129083,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280530,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41928,"kb_used_data":10736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329912,"statfs":{"total":96636764160,"available":96593829888,"internally_reserved":0,"allocated":10993664,"data_stored":15696189,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280530,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41412,"kb_used_data":10224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330428,"statfs":{"total":96636764160,"available":96594358272,"internally_reserved":0,"allocated":10469376,"data_stored":12603709,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40428,"kb_used_data":9240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331412,"statfs":{"total":96636764160,"available":96595365888,"internally_reserved":0,"allocated":9461760,"data_stored":12742973,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":9636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":9867264,"data_stored":13564392,"data_compressed":184366,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43260,"kb_used_data":12072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328580,"statfs":{"total":96636764160,"available":96592465920,"internally_reserved":0,"allocated":12361728,"data_stored":17207613,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280530,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40084,"kb_used_data":8896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331756,"statfs":{"total":96636764160,"available":96595718144,"internally_reserved":0,"allocated":9109504,"data_stored":13246781,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":6048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":6193152,"data_stored":7778640,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280530,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42568,"kb_used_data":11376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329272,"statfs":{"total":96636764160,"available":96593174528,"internally_reserved":0,"allocated":11649024,"data_stored":15466813,"data_compressed":181366,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10317824,"data_stored":15478784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10387456,"data_stored":14360576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7249920,"data_stored":9060352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10330112,"data_stored":14430208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10043392,"data_stored":15007744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9519104,"data_stored":11915264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":12054528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8904704,"data_stored":12865536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11411456,"data_stored":16519168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10743808,"data_stored":15048704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8159232,"data_stored":12558336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8884224,"data_stored":11444224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":5238784,"data_stored":7090176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":10043392,"data_stored":14069760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10698752,"data_stored":14778368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":11194368,"data_stored":14696448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:37.281 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T04:42:37.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:42:37.281 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T04:42:37.282 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-16T04:42:37.292 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:37.293 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:37.302 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:37.302 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:37.302 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:42:37.302 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:37.316 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:42:37.331 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:37.331 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:37.355 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:37.351+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:37.366 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:37.367 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:37.401 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:37.419 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:37.419 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:37.430 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:37.427+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:37.440 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:37.440 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:37.461 INFO:tasks.rados.rados.0.smithi165.stdout:309: done (1 left) 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:310: delete oid 48 current snap is 24 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:307: finishing write tid 6 to smithi16530038-25 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:307: oid 25 updating version 0 to 162 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:307: oid 25 version 162 is already newer than 160 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:307: oid 25 version 162 is already newer than 159 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:307: oid 25 version 162 is already newer than 161 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:307: oid 25 version 162 is already newer than 158 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 162 (ObjNum 117 snap 24 seq_num 117) dirty exists 2024-09-16T04:42:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:307: left oid 25 (ObjNum 117 snap 24 seq_num 117) 2024-09-16T04:42:37.463 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:37.463 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:37.482 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:37.482 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:37.483 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:37.483 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:37.497 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:42:37.532 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:37.527+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:37.594 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:37.594 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:37.614 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:37.615 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:37.615 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:37.615 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:37.623 INFO:teuthology.orchestra.run.smithi059.stdout:moved 0 / 270 (0%) 2024-09-16T04:42:37.623 INFO:teuthology.orchestra.run.smithi059.stdout:avg 16.875 2024-09-16T04:42:37.623 INFO:teuthology.orchestra.run.smithi059.stdout:stddev 2.68968 -> 2.68968 (expected baseline 3.97748) 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout:min osd.12 with 11 -> 11 pgs (0.651852 -> 0.651852 * mean) 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout:max osd.13 with 21 -> 21 pgs (1.24444 -> 1.24444 * mean) 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout:oload 110 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout:max_change 0.05 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout:max_change_osds 4 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout:average_utilization 0.0004 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stdout:overload_utilization 0.0005 2024-09-16T04:42:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:no change 2024-09-16T04:42:37.629 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:42:37.632 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:37.627+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:37.640 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:37.641 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:37.642 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T04:42:37.642 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-09-16T04:42:37.662 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:37.663 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:37.686 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:37.686 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:37.724 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:37.724 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:37.733 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:37.727+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:37.748 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:37.755 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:37.755 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:37.771 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:42:37.774 INFO:tasks.rados.rados.0.smithi165.stdout:307: done (1 left) 2024-09-16T04:42:37.774 INFO:tasks.rados.rados.0.smithi165.stdout:310: done (0 left) 2024-09-16T04:42:37.774 INFO:tasks.rados.rados.0.smithi165.stdout:311: copy_from oid 36 from oid 46 current snap is 24 2024-09-16T04:42:37.774 INFO:tasks.rados.rados.0.smithi165.stdout:312: rollback oid 15 current snap is 24 2024-09-16T04:42:37.774 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 15 2024-09-16T04:42:37.775 INFO:tasks.rados.rados.0.smithi165.stdout:313: setattr oid 14 current snap is 24 2024-09-16T04:42:37.782 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 50 current snap is 24 2024-09-16T04:42:37.783 INFO:tasks.rados.rados.0.smithi165.stdout:314: seq_num 120 ranges {1818624=475136,2293760=655360,2949120=622592,3571712=278528} 2024-09-16T04:42:37.817 INFO:tasks.rados.rados.0.smithi165.stdout:314: writing smithi16530038-50 from 1818624 to 2293760 tid 1 2024-09-16T04:42:37.826 INFO:tasks.rados.rados.0.smithi165.stdout:314: writing smithi16530038-50 from 2293760 to 2949120 tid 2 2024-09-16T04:42:37.834 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:37.827+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:37.834 INFO:tasks.rados.rados.0.smithi165.stdout:314: writing smithi16530038-50 from 2949120 to 3571712 tid 3 2024-09-16T04:42:37.837 INFO:tasks.rados.rados.0.smithi165.stdout:314: writing smithi16530038-50 from 3571712 to 3850240 tid 4 2024-09-16T04:42:37.837 INFO:tasks.rados.rados.0.smithi165.stdout:315: read oid 12 snap 22 2024-09-16T04:42:37.837 INFO:tasks.rados.rados.0.smithi165.stdout:315: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:42:37.837 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 20 current snap is 24 2024-09-16T04:42:37.837 INFO:tasks.rados.rados.0.smithi165.stdout:316: seq_num 121 ranges {1622016=491520,2113536=802816,2916352=720896,3637248=32768} 2024-09-16T04:42:37.856 INFO:tasks.rados.rados.0.smithi165.stdout:316: writing smithi16530038-20 from 1622016 to 2113536 tid 1 2024-09-16T04:42:37.862 INFO:tasks.rados.rados.0.smithi165.stdout:316: writing smithi16530038-20 from 2113536 to 2916352 tid 2 2024-09-16T04:42:37.870 INFO:tasks.rados.rados.0.smithi165.stdout:316: writing smithi16530038-20 from 2916352 to 3637248 tid 3 2024-09-16T04:42:37.870 INFO:tasks.rados.rados.0.smithi165.stdout:316: writing smithi16530038-20 from 3637248 to 3670016 tid 4 2024-09-16T04:42:37.870 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 38 current snap is 24 2024-09-16T04:42:37.870 INFO:tasks.rados.rados.0.smithi165.stdout:317: seq_num 122 ranges {0=507904,507904=425984,933888=442368,1376256=540672,1916928=147456} 2024-09-16T04:42:37.872 INFO:tasks.rados.rados.0.smithi165.stdout:317: writing smithi16530038-38 from 0 to 507904 tid 1 2024-09-16T04:42:37.875 INFO:tasks.rados.rados.0.smithi165.stdout:317: writing smithi16530038-38 from 507904 to 933888 tid 2 2024-09-16T04:42:37.879 INFO:tasks.rados.rados.0.smithi165.stdout:317: writing smithi16530038-38 from 933888 to 1376256 tid 3 2024-09-16T04:42:37.882 INFO:tasks.rados.rados.0.smithi165.stdout:317: writing smithi16530038-38 from 1376256 to 1916928 tid 4 2024-09-16T04:42:37.884 INFO:tasks.rados.rados.0.smithi165.stdout:317: writing smithi16530038-38 from 1916928 to 2064384 tid 5 2024-09-16T04:42:37.884 INFO:tasks.rados.rados.0.smithi165.stdout:318: rollback oid 4 current snap is 24 2024-09-16T04:42:37.884 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 4 to 23 2024-09-16T04:42:37.884 INFO:tasks.rados.rados.0.smithi165.stdout:319: snap_remove snap 21 2024-09-16T04:42:37.894 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:37.895 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:37.910 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:42:37.934 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:37.931+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:37.977 INFO:teuthology.orchestra.run.smithi059.stdout:moved 0 / 270 (0%) 2024-09-16T04:42:37.978 INFO:teuthology.orchestra.run.smithi059.stdout:avg 16.875 2024-09-16T04:42:37.978 INFO:teuthology.orchestra.run.smithi059.stdout:stddev 2.68968 -> 2.68968 (expected baseline 3.97748) 2024-09-16T04:42:37.978 INFO:teuthology.orchestra.run.smithi059.stdout:min osd.12 with 11 -> 11 pgs (0.651852 -> 0.651852 * mean) 2024-09-16T04:42:37.978 INFO:teuthology.orchestra.run.smithi059.stdout:max osd.13 with 21 -> 21 pgs (1.24444 -> 1.24444 * mean) 2024-09-16T04:42:37.978 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:37.978 INFO:teuthology.orchestra.run.smithi059.stdout:oload 1000 2024-09-16T04:42:37.978 INFO:teuthology.orchestra.run.smithi059.stdout:max_change 1 2024-09-16T04:42:37.979 INFO:teuthology.orchestra.run.smithi059.stdout:max_change_osds 4 2024-09-16T04:42:37.979 INFO:teuthology.orchestra.run.smithi059.stdout:average_utilization 0.0004 2024-09-16T04:42:37.979 INFO:teuthology.orchestra.run.smithi059.stdout:overload_utilization 0.0044 2024-09-16T04:42:37.979 INFO:teuthology.orchestra.run.smithi059.stderr:no change 2024-09-16T04:42:37.995 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T04:42:37.995 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-09-16T04:42:38.031 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:38.031 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:38.035 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:38.031+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:38.046 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:42:38.136 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:38.131+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:38.170 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:38.171 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:38.186 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:42:38.237 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:38.231+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:38.338 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:38.335+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:38.339 INFO:teuthology.orchestra.run.smithi059.stdout:moved 0 / 270 (0%) 2024-09-16T04:42:38.339 INFO:teuthology.orchestra.run.smithi059.stdout:avg 16.875 2024-09-16T04:42:38.339 INFO:teuthology.orchestra.run.smithi059.stdout:stddev 2.68968 -> 2.68968 (expected baseline 3.97748) 2024-09-16T04:42:38.339 INFO:teuthology.orchestra.run.smithi059.stdout:min osd.12 with 11 -> 11 pgs (0.651852 -> 0.651852 * mean) 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stdout:max osd.13 with 21 -> 21 pgs (1.24444 -> 1.24444 * mean) 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stdout:oload 110 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stdout:max_change 1 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stdout:max_change_osds 4 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stdout:average_utilization 0.0004 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stdout:overload_utilization 0.0005 2024-09-16T04:42:38.340 INFO:teuthology.orchestra.run.smithi059.stderr:no change 2024-09-16T04:42:38.353 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:38.353 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:38.357 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-16T04:42:38.357 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-09-16T04:42:38.367 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:42:38.441 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:38.439+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:319: done (8 left) 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:320: read oid 18 snap 22 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:320: expect (ObjNum 86 snap 14 seq_num 86) 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:321: snap_create 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:312: finishing rollback tid 0 to smithi16530038-15 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:312: finishing rollback tid 1 to smithi16530038-15 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 62 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing write tid 1 to smithi16530038-50 2024-09-16T04:42:38.470 INFO:tasks.rados.rados.0.smithi165.stdout:312: finishing rollback tid 2 to smithi16530038-15 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 61 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing write tid 1 to smithi16530038-38 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing write tid 2 to smithi16530038-38 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 1 to smithi16530038-20 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 2 to smithi16530038-20 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing write tid 2 to smithi16530038-50 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:311: finishing copy_from racing read to smithi16530038-36 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:318: finishing rollback tid 0 to smithi16530038-4 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:318: finishing rollback tid 1 to smithi16530038-4 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:318: finishing rollback tid 2 to smithi16530038-4 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 40 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:38.471 INFO:tasks.rados.rados.0.smithi165.stdout:315: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:42:38.487 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 3 to smithi16530038-20 2024-09-16T04:42:38.488 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing write tid 3 to smithi16530038-38 2024-09-16T04:42:38.488 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing write tid 3 to smithi16530038-50 2024-09-16T04:42:38.488 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing write tid 4 to smithi16530038-50 2024-09-16T04:42:38.488 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 4 to smithi16530038-20 2024-09-16T04:42:38.488 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing write tid 5 to smithi16530038-50 2024-09-16T04:42:38.488 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 5 to smithi16530038-20 2024-09-16T04:42:38.492 INFO:tasks.rados.rados.0.smithi165.stdout:311: finishing copy_from to smithi16530038-36 2024-09-16T04:42:38.492 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 20 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-16T04:42:38.498 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:38.498 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:38.513 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:42:38.541 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:38.539+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:38.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:42:38.592 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing write tid 4 to smithi16530038-38 2024-09-16T04:42:38.644 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:38.635+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:38.647 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:38.647 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:38.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:42:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 6 to smithi16530038-20 2024-09-16T04:42:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 20 updating version 0 to 79 2024-09-16T04:42:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 20 updating version 79 to 80 2024-09-16T04:42:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 20 version 80 is already newer than 78 2024-09-16T04:42:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 20 updating version 80 to 81 2024-09-16T04:42:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 20 updating version 81 to 82 2024-09-16T04:42:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 82 (ObjNum 121 snap 24 seq_num 121) dirty exists 2024-09-16T04:42:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:316: left oid 20 (ObjNum 121 snap 24 seq_num 121) 2024-09-16T04:42:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing write tid 5 to smithi16530038-38 2024-09-16T04:42:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing write tid 6 to smithi16530038-38 2024-09-16T04:42:38.712 INFO:teuthology.orchestra.run.smithi059.stdout:moved 0 / 270 (0%) 2024-09-16T04:42:38.712 INFO:teuthology.orchestra.run.smithi059.stdout:avg 16.875 2024-09-16T04:42:38.712 INFO:teuthology.orchestra.run.smithi059.stdout:stddev 2.68968 -> 2.68968 (expected baseline 3.97748) 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:min osd.12 with 11 -> 11 pgs (0.651852 -> 0.651852 * mean) 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:max osd.13 with 21 -> 21 pgs (1.24444 -> 1.24444 * mean) 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:oload 1000 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:max_change 0.05 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:max_change_osds 4 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:average_utilization 0.0004 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:overload_utilization 0.0044 2024-09-16T04:42:38.713 INFO:teuthology.orchestra.run.smithi059.stderr:no change 2024-09-16T04:42:38.730 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T04:42:38.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-16T04:42:38.744 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:38.739+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:38.788 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:38.789 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:38.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:42:38.844 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:38.839+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:38.879 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing write tid 6 to smithi16530038-50 2024-09-16T04:42:38.879 INFO:tasks.rados.rados.0.smithi165.stdout:314: oid 50 updating version 0 to 31 2024-09-16T04:42:38.879 INFO:tasks.rados.rados.0.smithi165.stdout:314: oid 50 updating version 31 to 33 2024-09-16T04:42:38.879 INFO:tasks.rados.rados.0.smithi165.stdout:314: oid 50 updating version 33 to 35 2024-09-16T04:42:38.879 INFO:tasks.rados.rados.0.smithi165.stdout:314: oid 50 version 35 is already newer than 34 2024-09-16T04:42:38.879 INFO:tasks.rados.rados.0.smithi165.stdout:314: oid 50 version 35 is already newer than 32 2024-09-16T04:42:38.879 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 35 (ObjNum 120 snap 24 seq_num 120) dirty exists 2024-09-16T04:42:38.880 INFO:tasks.rados.rados.0.smithi165.stdout:314: left oid 50 (ObjNum 120 snap 24 seq_num 120) 2024-09-16T04:42:38.910 INFO:tasks.rados.rados.0.smithi165.stdout:320: expect (ObjNum 86 snap 14 seq_num 86) 2024-09-16T04:42:38.925 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:38.926 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:38.940 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:42:38.945 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:38.943+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:39.007 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing write tid 7 to smithi16530038-38 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:317: oid 38 updating version 0 to 63 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:317: oid 38 updating version 63 to 66 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:317: oid 38 version 66 is already newer than 64 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:317: oid 38 updating version 66 to 68 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:317: oid 38 version 68 is already newer than 65 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:317: oid 38 version 68 is already newer than 67 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 68 (ObjNum 122 snap 24 seq_num 122) dirty exists 2024-09-16T04:42:39.008 INFO:tasks.rados.rados.0.smithi165.stdout:317: left oid 38 (ObjNum 122 snap 24 seq_num 122) 2024-09-16T04:42:39.018 INFO:teuthology.orchestra.run.smithi059.stdout:moved 0 / 270 (0%) 2024-09-16T04:42:39.018 INFO:teuthology.orchestra.run.smithi059.stdout:avg 16.875 2024-09-16T04:42:39.018 INFO:teuthology.orchestra.run.smithi059.stdout:stddev 2.68968 -> 2.68968 (expected baseline 3.97748) 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout:min osd.12 with 11 -> 11 pgs (0.651852 -> 0.651852 * mean) 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout:max osd.13 with 21 -> 21 pgs (1.24444 -> 1.24444 * mean) 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout:oload 110 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout:max_change 3 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout:max_change_osds 4 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout:average_utilization 0.0004 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stdout:overload_utilization 0.0005 2024-09-16T04:42:39.019 INFO:teuthology.orchestra.run.smithi059.stderr:no change 2024-09-16T04:42:39.046 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:39.043+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:39.064 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:39.064 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:39.079 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:42:39.147 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:39.143+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:39.197 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:39.197 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:39.206 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:39.199+0000 7f90016d3640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 17 after trimming was started 2024-09-16T04:42:39.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:42:39.248 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:39.243+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:39.349 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:39.343+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:39.382 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:39.382 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:39.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:42:39.450 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:39.443+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:39.467 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:311: done (9 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:312: done (8 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:313: done (7 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:314: done (6 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:315: done (5 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:316: done (4 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:317: done (3 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:318: done (2 left) 2024-09-16T04:42:39.473 INFO:tasks.rados.rados.0.smithi165.stdout:320: done (1 left) 2024-09-16T04:42:39.474 INFO:tasks.rados.rados.0.smithi165.stdout:321: done (0 left) 2024-09-16T04:42:39.474 INFO:tasks.rados.rados.0.smithi165.stdout:322: read oid 23 snap -1 2024-09-16T04:42:39.474 INFO:tasks.rados.rados.0.smithi165.stdout:322: expect (ObjNum 111 snap 20 seq_num 111) 2024-09-16T04:42:39.474 INFO:tasks.rados.rados.0.smithi165.stdout:323: snap_remove snap 11 2024-09-16T04:42:39.521 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:39.521 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:39.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:42:39.551 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:39.547+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:39.652 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:39.651+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:39.658 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:39.658 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:39.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:42:39.753 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:39.747+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:39.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:39.794 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:39.794 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:39.808 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:42:39.854 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:39.847+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:39.923 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:39.923 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:39.930 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:39.930 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:39.944 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:42:39.946 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:39.946 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:39.966 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:39.966 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:39.990 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:39.994 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:39.991+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:40.010 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:40.035 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:40.057 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:40.051+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:40.059 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:40.067 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:40.068 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:40.083 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:42:40.084 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:40.084 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:40.107 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:40.108 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:40.132 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:40.133 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:40.157 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:40.155+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:40.236 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:40.236 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:40.251 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:40.251 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:40.258 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:40.255+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:40.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:42:40.266 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:40.267 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:40.293 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:40.293 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:40.324 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:40.352 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:40.359 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:40.355+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:40.380 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:40.380 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:40.403 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:40.403 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:40.403 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:40.403 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:40.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:42:40.460 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:40.455+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:40.474 INFO:tasks.rados.rados.0.smithi165.stdout:323: done (1 left) 2024-09-16T04:42:40.474 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 45 current snap is 25 2024-09-16T04:42:40.474 INFO:tasks.rados.rados.0.smithi165.stdout:324: seq_num 123 ranges {1916928=524288,2441216=573440,3014656=688128,3702784=294912} 2024-09-16T04:42:40.493 INFO:tasks.rados.rados.0.smithi165.stdout:324: writing smithi16530038-45 from 1916928 to 2441216 tid 1 2024-09-16T04:42:40.497 INFO:tasks.rados.rados.0.smithi165.stdout:324: writing smithi16530038-45 from 2441216 to 3014656 tid 2 2024-09-16T04:42:40.502 INFO:tasks.rados.rados.0.smithi165.stdout:324: writing smithi16530038-45 from 3014656 to 3702784 tid 3 2024-09-16T04:42:40.505 INFO:tasks.rados.rados.0.smithi165.stdout:324: writing smithi16530038-45 from 3702784 to 3997696 tid 4 2024-09-16T04:42:40.505 INFO:tasks.rados.rados.0.smithi165.stdout:325: rollback oid 35 current snap is 25 2024-09-16T04:42:40.505 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 15 2024-09-16T04:42:40.505 INFO:tasks.rados.rados.0.smithi165.stdout:322: expect (ObjNum 111 snap 20 seq_num 111) 2024-09-16T04:42:40.543 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:40.543 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:40.543 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:40.544 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:40.544 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:40.552 INFO:tasks.rados.rados.0.smithi165.stdout:322: done (2 left) 2024-09-16T04:42:40.552 INFO:tasks.rados.rados.0.smithi165.stdout:326: snap_create 2024-09-16T04:42:40.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:42:40.560 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:40.555+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:40.662 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:40.659+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:40.683 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:40.683 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:40.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:40.683 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:40.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.683 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:122 3.as0 3:5619390f:::smithi16530038-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.000471544, 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51059284400000005, 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.684 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678088+0000", 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.12e-06 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678098+0000", 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0000000000000001e-05 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678119+0000", 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1676e-05 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678135+0000", 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5526999999999999e-05 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682759+0000", 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046242649999999998 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682887+0000", 2024-09-16T04:42:40.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012783599999999999 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.683968+0000", 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010808090000000001 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.784795+0000", 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10082761899999999 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188432+0000", 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40363656399999998 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188678+0000", 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024642799999999998 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.689 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:123 3.as0 3:5619390f:::smithi16530038-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.000340207999997, 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91283640799999999, 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 123 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678219+0000", 2024-09-16T04:42:40.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4030000000000001e-06 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678231+0000", 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678267+0000", 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5719000000000001e-05 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678291+0000", 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4006e-05 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684378+0000", 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060877320000000002 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684496+0000", 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118076 2024-09-16T04:42:40.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188804+0000", 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50430794400000001 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.189032+0000", 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000227235 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.591053+0000", 2024-09-16T04:42:40.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402021666 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:133 3.as0 3:5f4281cc:::smithi16530038-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.795152023, 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50755930599999999, 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883408+0000", 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8669999999999998e-06 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884235+0000", 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082694699999999997 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884270+0000", 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4690000000000002e-05 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884286+0000", 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6082999999999998e-05 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884441+0000", 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155852 2024-09-16T04:42:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884838+0000", 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000396494 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.886865+0000", 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020266709999999999 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965465+0000", 2024-09-16T04:42:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078600544999999994 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390790+0000", 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42532448499999997 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390964+0000", 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017467200000000001 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:134 3.as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.794240533, 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50704782699999995, 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884318+0000", 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884326+0000", 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.32e-06 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884339+0000", 2024-09-16T04:42:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.274e-05 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884348+0000", 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0229999999999998e-06 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888432+0000", 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040844899999999996 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888564+0000", 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131419 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889204+0000", 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064011899999999997 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965469+0000", 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076265854999999994 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391186+0000", 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.425716382 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391364+0000", 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017849599999999999 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:135 3.as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.794146437000002, 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0075776860000001, 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884412+0000", 2024-09-16T04:42:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3399999999999997e-07 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884416+0000", 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3059999999999999e-06 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884428+0000", 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2500000000000001e-05 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884439+0000", 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0662999999999999e-05 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889599+0000", 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051597179999999998 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889722+0000", 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012330300000000001 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392670+0000", 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50294774399999997 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392905+0000", 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023530799999999999 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.891988+0000", 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49908321 2024-09-16T04:42:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:176 3.as0 3:5619390f:::smithi16530038-41:head [copy-from ver 10] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.603859659000001, 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71635610400000005, 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074701+0000", 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9500000000000003e-06 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074716+0000", 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4523e-05 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074742+0000", 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5795999999999998e-05 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074764+0000", 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2646999999999998e-05 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074937+0000", 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017243599999999999 2024-09-16T04:42:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074957+0000", 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9873000000000001e-05 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178498+0000", 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103540929 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178630+0000", 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132399 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.493151+0000", 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.3145212 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.597069+0000", 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103917643 2024-09-16T04:42:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.790921+0000", 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19385244300000001 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791054+0000", 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013226500000000001 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:177 3.as0 3:5619390f:::smithi16530038-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.603740125999998, 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42059460999999998, 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177 2024-09-16T04:42:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074819+0000", 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074828+0000", 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7399999999999993e-06 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074850+0000", 2024-09-16T04:42:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2802999999999998e-05 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074873+0000", 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2180000000000001e-05 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075093+0000", 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000220645 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075109+0000", 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5823000000000001e-05 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178898+0000", 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10378908100000001 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178962+0000", 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489000000000001e-05 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495097+0000", 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31613566599999998 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495230+0000", 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133039 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495412+0000", 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018130800000000001 2024-09-16T04:42:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:194 3.as0 3:5f4281cc:::smithi16530038-45:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.578581094, 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59800688199999996, 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 194 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099979+0000", 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6400000000000001e-06 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099990+0000", 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.15e-05 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100022+0000", 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1569999999999998e-05 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100039+0000", 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7486000000000002e-05 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179005+0000", 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078965628999999996 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179189+0000", 2024-09-16T04:42:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184155 2024-09-16T04:42:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.486772+0000", 2024-09-16T04:42:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30758345500000001 2024-09-16T04:42:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495583+0000", 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088103490000000003 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697796+0000", 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20221325300000001 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697983+0000", 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018684499999999999 2024-09-16T04:42:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:276 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.579777610000001, 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71405619499999995, 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 276 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098783+0000", 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098798+0000", 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5213e-05 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098845+0000", 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6853e-05 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098872+0000", 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6622999999999999e-05 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099054+0000", 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018249499999999999 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099075+0000", 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.065e-05 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207663+0000", 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858786400000001 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207828+0000", 2024-09-16T04:42:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016534199999999999 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812836+0000", 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60500746100000002 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:277 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.579629758000003, 2024-09-16T04:42:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81467523799999997, 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 277 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098929+0000", 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4670000000000001e-06 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098938+0000", 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098964+0000", 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5979999999999999e-05 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098980+0000", 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5917e-05 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099168+0000", 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000188228 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099176+0000", 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.307912+0000", 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20873598700000001 2024-09-16T04:42:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.308068+0000", 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156448 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913603+0000", 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60553407699999995 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:278 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.579553648000001, 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81515018400000006, 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 278 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099005+0000", 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-06 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099012+0000", 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9e-06 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099031+0000", 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9303000000000001e-05 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099046+0000", 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5003e-05 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099213+0000", 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000166773 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099218+0000", 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:42:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408464+0000", 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30924607700000001 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408612+0000", 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014793900000000001 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.914154+0000", 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50554194200000002 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.as0 43/32 ECSubRead(tid=39, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.269636818999999, 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40299000499999998, 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408986+0000", 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409043+0000", 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6589999999999999e-05 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811767+0000", 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40272446200000001 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811910+0000", 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014330300000000001 2024-09-16T04:42:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=37, attrs_read=0))", 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.269237498000003, 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.403543547, 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409389+0000", 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9126000000000002e-05 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409407+0000", 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8097e-05 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812374+0000", 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40296689699999999 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812863+0000", 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048942699999999996 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.268877041000003, 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40266660700000001, 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409680+0000", 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409681+0000", 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2699999999999998e-07 2024-09-16T04:42:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409710+0000", 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9036000000000001e-05 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409721+0000", 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1187e-05 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409729+0000", 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7260000000000002e-06 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812298+0000", 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40256918800000002 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812347+0000", 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8542999999999999e-05 2024-09-16T04:42:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.268800941999999, 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402371019, 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409756+0000", 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409757+0000", 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409802+0000", 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4969999999999998e-05 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409813+0000", 2024-09-16T04:42:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1127e-05 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409820+0000", 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4499999999999998e-06 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812023+0000", 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40220215599999998 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812127+0000", 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104549 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.268634263000003, 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40309429400000002, 2024-09-16T04:42:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409924+0000", 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409963+0000", 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8753e-05 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409981+0000", 2024-09-16T04:42:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8737e-05 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409991+0000", 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3889999999999994e-06 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812922+0000", 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40293080100000001 2024-09-16T04:42:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.813017+0000", 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5457000000000004e-05 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=38, attrs_read=0))", 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.167553996000002, 2024-09-16T04:42:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40262853199999998, 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511003+0000", 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511049+0000", 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5912999999999999e-05 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.511068+0000", 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.855e-05 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913200+0000", 2024-09-16T04:42:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40213229900000003 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913632+0000", 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043176999999999999 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:704 3.14s0 3:28a14731:::smithi16530038-3:head [rollback d] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:42:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.017321071, 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40748865899999998, 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 704 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661237+0000", 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2100000000000001e-06 2024-09-16T04:42:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661243+0000", 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0970000000000002e-06 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661259+0000", 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5622999999999999e-05 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661270+0000", 2024-09-16T04:42:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1182999999999999e-05 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.763754+0000", 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102483703 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.763924+0000", 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000170182 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.765124+0000", 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001200191 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.767658+0000", 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025341579999999999 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068502+0000", 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30084363400000003 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068725+0000", 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000222678 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:705 3.14s0 3:28a14731:::smithi16530038-3:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.017240864, 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40781490599999998, 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.779 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 705 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661317+0000", 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2729999999999999e-06 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661324+0000", 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3629999999999999e-06 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661335+0000", 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1396999999999999e-05 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661346+0000", 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0546e-05 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.766069+0000", 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104723092 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.766222+0000", 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015317900000000001 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.767028+0000", 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080575099999999995 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.868139+0000", 2024-09-16T04:42:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101110693 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068984+0000", 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20084581600000001 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.069131+0000", 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014679600000000001 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:748 3.14s0 3:2b4d2b94:::smithi16530038-49:head [stat,delete] snapc 19=[19,18,17,16,11,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.2331946250000003, 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.406000319, 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:40.785 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 748 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445364+0000", 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445369+0000", 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4140000000000002e-06 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445388+0000", 2024-09-16T04:42:40.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8292999999999999e-05 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445402+0000", 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4620000000000001e-05 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445575+0000", 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017319199999999999 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445594+0000", 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8363e-05 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.651371+0000", 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20577725199999999 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.651555+0000", 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018435899999999999 2024-09-16T04:42:40.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.653237+0000", 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001681491 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.657570+0000", 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043331439999999997 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:40.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.851129+0000", 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19355888900000001 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.851363+0000", 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023383200000000001 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.791 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:40.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:42:40.794 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:40.787+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:40.893 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:40.891+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:40.925 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.1as5 90/69 ECSubWrite(tid=116, reqid=osd.14.0:116, at_version=90'60, trim_to=0'0, roll_forward_to=87'53))", 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:40.803127+0000", 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.122231722, 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12224861300000001, 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.926 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.803127+0000", 2024-09-16T04:42:40.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.803127+0000", 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.803128+0000", 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5600000000000001e-06 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.803200+0000", 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1123000000000003e-05 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.803273+0000", 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3206000000000003e-05 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903368+0000", 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100094853 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903443+0000", 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5895000000000004e-05 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903452+0000", 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.930 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.12s1 90/53 ECSubWrite(tid=27, reqid=osd.9.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=90'28))", 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:40.903597+0000", 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.021762048999999999, 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021817361, 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903597+0000", 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903597+0000", 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903598+0000", 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3540000000000001e-06 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903622+0000", 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4556e-05 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903642+0000", 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9729999999999999e-05 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903654+0000", 2024-09-16T04:42:40.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.208e-05 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903934+0000", 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028008200000000002 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:40.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.903945+0000", 2024-09-16T04:42:40.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0771000000000001e-05 2024-09-16T04:42:40.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.935 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 90/67 ECSubWriteReply(tid=202, last_complete=90'180, committed=1, applied=1))", 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:40.905621+0000", 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.019738024, 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019826053999999999, 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905621+0000", 2024-09-16T04:42:40.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905621+0000", 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905622+0000", 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5340000000000001e-06 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905635+0000", 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2765999999999999e-05 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905661+0000", 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5914e-05 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905673+0000", 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1863e-05 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 90/67 ECSubWriteReply(tid=202, last_complete=90'180, committed=1, applied=1))", 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:40.905878+0000", 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.019480821999999998, 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019593260000000001, 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905878+0000", 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905878+0000", 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905879+0000", 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9999999999999995e-07 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905887+0000", 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5830000000000007e-06 2024-09-16T04:42:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905902+0000", 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4413000000000001e-05 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.905911+0000", 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8899999999999996e-06 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 90/67 ECSubWriteReply(tid=203, last_complete=90'182, committed=1, applied=1))", 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:40.908523+0000", 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.016835725999999999, 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016972886999999999, 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908523+0000", 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908523+0000", 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908524+0000", 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2300000000000001e-06 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908530+0000", 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0000000000000002e-06 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908553+0000", 2024-09-16T04:42:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2784000000000001e-05 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908565+0000", 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-05 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 90/67 ECSubWriteReply(tid=204, last_complete=90'184, committed=1, applied=1))", 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:40.908991+0000", 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.016367528999999999, 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016545913999999998, 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908991+0000", 2024-09-16T04:42:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908991+0000", 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908992+0000", 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9400000000000004e-07 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.908996+0000", 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4229999999999998e-06 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.909013+0000", 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6693000000000001e-05 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:40.909023+0000", 2024-09-16T04:42:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5859999999999994e-06 2024-09-16T04:42:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T04:42:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 6 2024-09-16T04:42:40.953 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:40.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:40.994 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:40.991+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:41.017 INFO:tasks.rados.rados.0.smithi165.stdout:325: finishing rollback tid 0 to smithi16530038-35 2024-09-16T04:42:41.097 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:41.091+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:41.107 INFO:tasks.rados.rados.0.smithi165.stdout:324: finishing write tid 1 to smithi16530038-45 2024-09-16T04:42:41.125 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:41.125 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:41.125 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:41.125 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:41.126 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:41.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:41.197 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:41.191+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:41.207 INFO:tasks.rados.rados.0.smithi165.stdout:324: finishing write tid 2 to smithi16530038-45 2024-09-16T04:42:41.207 INFO:tasks.rados.rados.0.smithi165.stdout:324: finishing write tid 3 to smithi16530038-45 2024-09-16T04:42:41.208 INFO:tasks.rados.rados.0.smithi165.stdout:324: finishing write tid 4 to smithi16530038-45 2024-09-16T04:42:41.298 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:41.291+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:142 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.127648794000002, 2024-09-16T04:42:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98845964500000005, 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 142 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175562+0000", 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6839999999999998e-06 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175978+0000", 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041531000000000001 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176014+0000", 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5913e-05 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176030+0000", 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5872999999999999e-05 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658368+0000", 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.48233846800000002 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658643+0000", 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000274651 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.659966+0000", 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013230399999999999 2024-09-16T04:42:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665031+0000", 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050652290000000001 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.163900+0000", 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.498868961 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164018+0000", 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118516 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:143 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.121170432, 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.982808864, 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182041+0000", 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5099999999999999e-06 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182677+0000", 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063672200000000005 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182720+0000", 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2339999999999998e-05 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182742+0000", 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1929999999999998e-05 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661153+0000", 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.478411009 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661298+0000", 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145659 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.662610+0000", 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001311967 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766105+0000", 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10349428300000001 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164694+0000", 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39858998099999998 2024-09-16T04:42:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164846+0000", 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015146300000000001 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:144 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.120430611000003, 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98240055400000004, 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 144 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182779+0000", 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182787+0000", 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.977e-06 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182810+0000", 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3543000000000001e-05 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182839+0000", 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9e-05 2024-09-16T04:42:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663652+0000", 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.480813243 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663795+0000", 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143108 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.664540+0000", 2024-09-16T04:42:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074409499999999996 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766192+0000", 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10165231600000001 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165013+0000", 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39882149300000003 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165177+0000", 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164146 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:145 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.120345747999998, 2024-09-16T04:42:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.385587095, 2024-09-16T04:42:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182863+0000", 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3039999999999999e-06 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182868+0000", 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:42:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182891+0000", 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2362999999999999e-05 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182911+0000", 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9813000000000002e-05 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665100+0000", 2024-09-16T04:42:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.482189849 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665304+0000", 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020365800000000001 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264646+0000", 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59934227699999998 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264794+0000", 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147412 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.568449+0000", 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30365514599999999 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:1 3.ds0 3:bc30766e:::smithi16530038-11:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.090816713999999, 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0541568889999999, 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212394+0000", 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9000000000000002e-06 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212406+0000", 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2289999999999999e-05 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212428+0000", 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2410000000000001e-05 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212462+0000", 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3708999999999999e-05 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765550+0000", 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55308802099999999 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765692+0000", 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014137199999999999 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.266548+0000", 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50085618700000001 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:172 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.004004778000002, 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57524237600000006, 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 172 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299206+0000", 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6859999999999999e-06 2024-09-16T04:42:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299276+0000", 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9592999999999996e-05 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299323+0000", 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6749999999999998e-05 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299347+0000", 2024-09-16T04:42:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4416000000000001e-05 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469676+0000", 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.170329287 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469835+0000", 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159006 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.874445+0000", 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40460963799999999 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:173 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.003820689999998, 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57613007299999996, 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.340 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299389+0000", 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7099999999999999e-06 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299399+0000", 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0866000000000001e-05 2024-09-16T04:42:41.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299427+0000", 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7117000000000001e-05 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299488+0000", 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1186000000000007e-05 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470324+0000", 2024-09-16T04:42:41.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17083640999999999 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470484+0000", 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015948199999999999 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.875517+0000", 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40503330199999998 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:174 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.003691910999997, 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67607313300000005, 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.345 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299517+0000", 2024-09-16T04:42:41.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.393e-06 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299526+0000", 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8230000000000001e-06 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299553+0000", 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6889999999999998e-05 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299573+0000", 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9944000000000001e-05 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471098+0000", 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.171524865 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471221+0000", 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012369900000000001 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.975589+0000", 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50436751899999999 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:282 3.ds0 3:b100bb67:::smithi16530038-42:head [copy-from ver 3] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 48.204310135, 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62041838900000001, 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.349 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 282 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098900+0000", 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6630000000000002e-06 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098911+0000", 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0447e-05 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098945+0000", 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4418999999999997e-05 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098960+0000", 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5119999999999999e-05 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099123+0000", 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016297600000000001 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099133+0000", 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0125999999999999e-05 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.303007+0000", 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20387381399999999 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.303177+0000", 2024-09-16T04:42:41.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016986900000000001 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.615144+0000", 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31196737099999999 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.620953+0000", 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058083869999999999 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.719092+0000", 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098139689000000002 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.719316+0000", 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000223508 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:283 3.ds0 3:b100bb67:::smithi16530038-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.099001+0000", 2024-09-16T04:42:41.355 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 48.204206583000001, 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51969615000000002, 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 283 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099001+0000", 2024-09-16T04:42:41.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099001+0000", 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.584e-06 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099011+0000", 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2630000000000003e-06 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099033+0000", 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2427000000000001e-05 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099049+0000", 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6033e-05 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099243+0000", 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019395799999999999 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099260+0000", 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6900000000000001e-05 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.403620+0000", 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30435966399999997 2024-09-16T04:42:41.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.403709+0000", 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9553000000000001e-05 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.618382+0000", 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.214672997 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.618533+0000", 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015042300000000001 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.618697+0000", 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016434800000000001 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:513 3.ds0 3:b56b9d63:::smithi16530038-2:head [append 0~720896] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:41.361 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.074832367999999, 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112358499999997, 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 513 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:41.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228379+0000", 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5269999999999999e-06 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.229180+0000", 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080095700000000004 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.229216+0000", 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6303000000000001e-05 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329324+0000", 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100107881 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329465+0000", 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140796 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329663+0000", 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019788499999999999 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.332137+0000", 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024749519999999999 2024-09-16T04:42:41.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437411+0000", 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105273296 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839350+0000", 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40193957600000002 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839499+0000", 2024-09-16T04:42:41.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000148412 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:514 3.ds0 3:b56b9d63:::smithi16530038-2:head [append 0~442368] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.329378+0000", 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.973829314, 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51044876699999997, 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 514 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329378+0000", 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329378+0000", 2024-09-16T04:42:41.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329380+0000", 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329910+0000", 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00052939999999999997 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329934+0000", 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3853e-05 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329949+0000", 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5167000000000001e-05 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.333951+0000", 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040021029999999999 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.334165+0000", 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021436200000000001 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.336388+0000", 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022231299999999998 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437417+0000", 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101028405 2024-09-16T04:42:41.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839745+0000", 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402328252 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839827+0000", 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2004999999999998e-05 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:515 3.ds0 3:b56b9d63:::smithi16530038-2:head [append 0~327680] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.329977+0000", 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.973230718, 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51048841099999998, 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 515 2024-09-16T04:42:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329977+0000", 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329977+0000", 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329978+0000", 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1599999999999999e-06 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330336+0000", 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035811099999999999 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330351+0000", 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4473e-05 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330360+0000", 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2329999999999995e-06 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.337819+0000", 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074591509999999998 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.338063+0000", 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000243761 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.340407+0000", 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002344793 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437420+0000", 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097012878999999996 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.840314+0000", 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40289380499999999 2024-09-16T04:42:41.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.840465+0000", 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015104499999999999 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:517 3.ds0 3:b56b9d63:::smithi16530038-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.972792441999999, 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71224451200000005, 2024-09-16T04:42:41.378 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 517 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330416+0000", 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330420+0000", 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.557e-06 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330431+0000", 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1223000000000001e-05 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330438+0000", 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7170000000000008e-06 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.841180+0000", 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51074163900000003 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.841322+0000", 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141433 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.042660+0000", 2024-09-16T04:42:41.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20133805299999999 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:568 3.ds0 3:b100bb67:::smithi16530038-42:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.401694281000001, 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60988380600000003, 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.383 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 568 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901535+0000", 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1696999999999999e-05 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901549+0000", 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.364e-05 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901580+0000", 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1335999999999999e-05 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901639+0000", 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8776000000000001e-05 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.300597+0000", 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.398958549 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.300848+0000", 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025097800000000001 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.302460+0000", 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016119750000000001 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.306368+0000", 2024-09-16T04:42:41.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039082070000000003 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.511231+0000", 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.204862715 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.511397+0000", 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000165933 2024-09-16T04:42:41.388 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:569 3.ds0 3:b100bb67:::smithi16530038-42:head [rollback 11] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.401457969999999, 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80813885399999996, 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.389 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 569 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:41.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901751+0000", 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6670000000000001e-06 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901804+0000", 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2306999999999999e-05 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901826+0000", 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2186000000000001e-05 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901845+0000", 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8802999999999999e-05 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.409423+0000", 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.507578164 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.409660+0000", 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023717500000000001 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.410803+0000", 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011429249999999999 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.611780+0000", 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20097747999999999 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.709760+0000", 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097980190999999994 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.709888+0000", 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012795600000000001 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:570 3.ds0 3:b100bb67:::smithi16530038-42:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.401333634, 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81140166199999997, 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 570 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901875+0000", 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.483e-06 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901883+0000", 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.357e-06 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901901+0000", 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8593e-05 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901917+0000", 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.556e-05 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.509597+0000", 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60768048500000005 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.509774+0000", 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000176909 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.510566+0000", 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079207799999999999 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.611782+0000", 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101216059 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.713154+0000", 2024-09-16T04:42:41.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10137199199999999 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.713276+0000", 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121146 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:754 3.ds0 3:b743ddf6:::smithi16530038-25:head [setxattr _header (57) in=7b] snapc 19=[19,18,17,16,11,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:36.450830+0000", 2024-09-16T04:42:41.400 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8523774199999998, 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70760600399999996, 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 754 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450830+0000", 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450830+0000", 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450831+0000", 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2240000000000001e-06 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450837+0000", 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7100000000000004e-06 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450851+0000", 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4156e-05 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450861+0000", 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9769999999999995e-06 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.658220+0000", 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20735928200000001 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.658396+0000", 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017554499999999999 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.659362+0000", 2024-09-16T04:42:41.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00096583300000000003 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.662442+0000", 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030804130000000002 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158286+0000", 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49584404500000001 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158436+0000", 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014981900000000001 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:755 3.ds0 3:b743ddf6:::smithi16530038-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:36.450893+0000", 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8523143300000005, 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71187130499999995, 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.406 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 755 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450893+0000", 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450893+0000", 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450894+0000", 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9300000000000006e-07 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450898+0000", 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1069999999999998e-06 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450911+0000", 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2799999999999999e-05 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450920+0000", 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.659849+0000", 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20892882600000001 2024-09-16T04:42:41.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.659990+0000", 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141056 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158659+0000", 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49866967000000001 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158953+0000", 2024-09-16T04:42:41.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029351399999999998 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.162765+0000", 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003811629 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:756 3.ds0 3:b100bb67:::smithi16530038-42:head [stat,delete] snapc 19=[19,18,17,16,11,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:36.450932+0000", 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8522752100000002, 2024-09-16T04:42:41.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70945773899999998, 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 756 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450932+0000", 2024-09-16T04:42:41.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450932+0000", 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450933+0000", 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2699999999999998e-07 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450937+0000", 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6430000000000001e-06 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450950+0000", 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2989999999999999e-05 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450957+0000", 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4460000000000004e-06 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.660125+0000", 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.209168248 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.660286+0000", 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016025300000000001 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.661434+0000", 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011481040000000001 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.663261+0000", 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018268659999999999 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.160204+0000", 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49694368999999999 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.160390+0000", 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018567200000000001 2024-09-16T04:42:41.416 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.417 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.417 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.417 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:41.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:41.419 INFO:tasks.rados.rados.0.smithi165.stdout:325: finishing rollback tid 1 to smithi16530038-35 2024-09-16T04:42:41.419 INFO:tasks.rados.rados.0.smithi165.stdout:325: finishing rollback tid 2 to smithi16530038-35 2024-09-16T04:42:41.419 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 33 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T04:42:41.419 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:41.411+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:41.494 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:42:41.496 INFO:tasks.rados.rados.0.smithi165.stdout:325: done (2 left) 2024-09-16T04:42:41.496 INFO:tasks.rados.rados.0.smithi165.stdout:326: done (1 left) 2024-09-16T04:42:41.496 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 43 current snap is 26 2024-09-16T04:42:41.496 INFO:tasks.rados.rados.0.smithi165.stdout:327: seq_num 124 ranges {2048000=540672,2588672=720896,3309568=786432,4096000=49152} 2024-09-16T04:42:41.517 INFO:tasks.rados.rados.0.smithi165.stdout:327: writing smithi16530038-43 from 2048000 to 2588672 tid 1 2024-09-16T04:42:41.519 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:41.515+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:41.523 INFO:tasks.rados.rados.0.smithi165.stdout:327: writing smithi16530038-43 from 2588672 to 3309568 tid 2 2024-09-16T04:42:41.529 INFO:tasks.rados.rados.0.smithi165.stdout:327: writing smithi16530038-43 from 3309568 to 4096000 tid 3 2024-09-16T04:42:41.530 INFO:tasks.rados.rados.0.smithi165.stdout:327: writing smithi16530038-43 from 4096000 to 4145152 tid 4 2024-09-16T04:42:41.530 INFO:tasks.rados.rados.0.smithi165.stdout:328: read oid 20 snap -1 2024-09-16T04:42:41.530 INFO:tasks.rados.rados.0.smithi165.stdout:328: expect (ObjNum 121 snap 24 seq_num 121) 2024-09-16T04:42:41.530 INFO:tasks.rados.rados.0.smithi165.stdout:329: snap_remove snap 15 2024-09-16T04:42:41.592 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:41.592 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:41.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.1as6 90/69 ECSubWrite(tid=120, reqid=client.4728.0:803, at_version=0'0, trim_to=0'0, roll_forward_to=90'61))", 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:41.516631+0000", 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.075526890999999999, 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.075569804000000004, 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "no flag points reached", 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:41.516631+0000", 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:41.516631+0000", 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:41.516632+0000", 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8899999999999998e-07 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:41.516645+0000", 2024-09-16T04:42:41.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2228e-05 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:41.516662+0000", 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7105e-05 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.596 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T04:42:41.596 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-16T04:42:41.596 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:41.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:41.619 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:41.615+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:41.720 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:41.715+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:41.780 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:41.780 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:41.780 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:41.780 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:41.781 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:41.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:41.822 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:41.815+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:41.921 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:41.921 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:41.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:41.921 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:154 3.fs0 3:f6276c08:::smithi16530038-20:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.70898742, 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2626387860000001, 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 154 2024-09-16T04:42:41.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210502+0000", 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9139999999999999e-06 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210519+0000", 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6969999999999998e-05 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210555+0000", 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6099000000000001e-05 2024-09-16T04:42:41.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210573+0000", 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7770000000000001e-05 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210738+0000", 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016486499999999999 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210962+0000", 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022410600000000001 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.269739+0000", 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.058777538 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272567+0000", 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028276550000000001 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.472907+0000", 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20033962699999999 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.473136+0000", 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000229242 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.927 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:155 3.fs0 3:f6276c08:::smithi16530038-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.708871410999997, 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.061292586, 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 155 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210614+0000", 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7000000000000003e-07 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210619+0000", 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210636+0000", 2024-09-16T04:42:41.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7436999999999999e-05 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210645+0000", 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.816e-06 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211299+0000", 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065386899999999996 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:41.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211385+0000", 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5852000000000004e-05 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271332+0000", 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.05994666 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271584+0000", 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025255399999999999 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271906+0000", 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032151100000000002 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:169 3.cs0 3:34145aa0:::smithi16530038-26:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.022807794000002, 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87942165800000005, 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 169 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896679+0000", 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3070000000000001e-06 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896692+0000", 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2785999999999999e-05 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896719+0000", 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7540000000000001e-05 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896735+0000", 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5897e-05 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998807+0000", 2024-09-16T04:42:41.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102071833 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998930+0000", 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012282600000000001 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.572766+0000", 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57383636000000005 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574732+0000", 2024-09-16T04:42:41.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019661589999999999 2024-09-16T04:42:41.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.775926+0000", 2024-09-16T04:42:41.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20119392799999999 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.776098+0000", 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172022 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.939 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:170 3.cs0 3:34145aa0:::smithi16530038-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.022721873999998, 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67734535900000004, 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896763+0000", 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896764+0000", 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-16T04:42:41.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896768+0000", 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896780+0000", 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2003e-05 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896791+0000", 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0876000000000001e-05 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101155+0000", 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20436343900000001 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.101252+0000", 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7195999999999996e-05 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573831+0000", 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.47257916700000002 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.573966+0000", 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134816 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574108+0000", 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014204199999999999 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:207 3.fs0 3:f1175b03:::smithi16530038-29:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 53.819076225000003, 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.69161456600000004, 2024-09-16T04:42:41.945 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 207 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100411+0000", 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100423+0000", 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.202e-05 2024-09-16T04:42:41.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100454+0000", 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0895999999999997e-05 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100469+0000", 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5614e-05 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174514+0000", 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.074044911000000005 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174748+0000", 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023420200000000001 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.687971+0000", 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51322260399999997 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690498+0000", 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002526978 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791786+0000", 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10128820500000001 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.792023+0000", 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000236772 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.950 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:208 3.fs0 3:f1175b03:::smithi16530038-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 53.818985879000003, 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.58977269499999996, 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.951 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100499+0000", 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100500+0000", 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100507+0000", 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100524+0000", 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.607e-05 2024-09-16T04:42:41.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100532+0000", 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8270000000000004e-06 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274070+0000", 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17353716399999999 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:41.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.274196+0000", 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012600600000000001 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689716+0000", 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41552020499999998 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.689979+0000", 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026336099999999999 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690271+0000", 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029222200000000002 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.7.22:2 3.cs0 3:34145aa0:::smithi16530038-26:head [assert-version v13,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.600063280999997, 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61025086299999998, 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.956 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319421+0000", 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319424+0000", 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319433+0000", 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4239999999999999e-06 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319460+0000", 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6965999999999999e-05 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.319498+0000", 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7518999999999998e-05 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.421237+0000", 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101738922 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.421547+0000", 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000310871 2024-09-16T04:42:41.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.518325+0000", 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096777742 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.518478+0000", 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015335199999999999 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:54.929672+0000", 2024-09-16T04:42:41.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41119360100000002 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:356 3.fs0 3:f6276c08:::smithi16530038-20:head [append 0~0] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.661949806000003, 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50700994099999996, 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.961 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 356 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257535+0000", 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257538+0000", 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3170000000000002e-06 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257552+0000", 2024-09-16T04:42:41.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.381e-05 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257588+0000", 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6046e-05 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357861+0000", 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10027301499999999 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458138+0000", 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100277112 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458411+0000", 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027305100000000001 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.460377+0000", 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019655160000000001 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465044+0000", 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046678409999999998 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.764419+0000", 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29937459100000002 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.764545+0000", 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125642 2024-09-16T04:42:41.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:357 3.fs0 3:f6276c08:::smithi16530038-20:head [rollback 3] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.561528385000003, 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60733727699999995, 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 357 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357956+0000", 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357959+0000", 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6630000000000002e-06 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.357971+0000", 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2660000000000001e-05 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.358002+0000", 2024-09-16T04:42:41.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0793000000000001e-05 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458105+0000", 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100102386 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.462250+0000", 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004145213 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.462405+0000", 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015478599999999999 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463262+0000", 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085764999999999995 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465233+0000", 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019711049999999999 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.965176+0000", 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.499942461 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.965293+0000", 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011756000000000001 2024-09-16T04:42:41.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:358 3.fs0 3:f6276c08:::smithi16530038-20:head [append 0~0] snapc 6=[6,3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.461349333000001, 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60770323100000001, 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 358 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458135+0000", 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458137+0000", 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0229999999999999e-06 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458148+0000", 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0784e-05 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458166+0000", 2024-09-16T04:42:41.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8116000000000001e-05 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.458182+0000", 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5424e-05 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463813+0000", 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056313439999999999 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.463964+0000", 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105300000000001 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.464741+0000", 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077701400000000002 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.465465+0000", 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072419100000000001 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.065719+0000", 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60025337300000003 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.065838+0000", 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011990899999999999 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.978 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:376 3.cs0 3:376489e5:::smithi16530038-34:head [delete] snapc 6=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.651533380000004, 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.52363831800000005, 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 376 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267951+0000", 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267954+0000", 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5339999999999998e-06 2024-09-16T04:42:41.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267963+0000", 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6700000000000006e-06 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.267992+0000", 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8646e-05 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268013+0000", 2024-09-16T04:42:41.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0814000000000001e-05 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268212+0000", 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000199681 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.268360+0000", 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014787599999999999 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.269584+0000", 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012233210000000001 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.271521+0000", 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373459999999999 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.791438+0000", 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51991722100000004 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:59.791589+0000", 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151209 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:424 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~573440] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.684552259, 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.502816084, 2024-09-16T04:42:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 424 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234932+0000", 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.234936+0000", 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.456e-06 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.235611+0000", 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00067546800000000003 2024-09-16T04:42:41.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.235647+0000", 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5466999999999998e-05 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335792+0000", 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10014563999999999 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.335931+0000", 2024-09-16T04:42:41.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000138582 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336098+0000", 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167465 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.338085+0000", 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019867890000000001 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737213+0000", 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39912832100000001 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737613+0000", 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039929100000000002 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.737748+0000", 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013560499999999999 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.989 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:426 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~344064] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.583094877000001, 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90840981600000004, 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 426 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336391+0000", 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0699999999999996e-07 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336788+0000", 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039739000000000002 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336805+0000", 2024-09-16T04:42:41.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6583999999999999e-05 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336814+0000", 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4299999999999995e-06 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739098+0000", 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40228381099999999 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739254+0000", 2024-09-16T04:42:41.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156506 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.740433+0000", 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011782450000000001 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943229+0000", 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20279692899999999 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244590+0000", 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30136072600000002 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244799+0000", 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020928799999999999 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:427 3.cs0 3:3dc0a373:::smithi16530038-10:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:41.995 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.582653809999996, 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90837145699999999, 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 427 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336832+0000", 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7400000000000002e-07 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336837+0000", 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9300000000000002e-06 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336857+0000", 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-05 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336890+0000", 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2852000000000003e-05 2024-09-16T04:42:41.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841600+0000", 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50471004399999997 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841814+0000", 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000214589 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.842536+0000", 2024-09-16T04:42:41.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072138400000000004 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943303+0000", 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10076771499999999 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245048+0000", 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30174494299999999 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245202+0000", 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015384599999999999 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:428 3.cs0 3:3dc0a373:::smithi16530038-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.582581703999999, 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1109408999999999, 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.001 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 428 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336904+0000", 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336920+0000", 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6067e-05 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336933+0000", 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336941+0000", 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4600000000000003e-06 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245817+0000", 2024-09-16T04:42:42.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90887578899999999 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.246090+0000", 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000273071 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.447844+0000", 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20175383 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:511 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~1458176,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.647249911999999, 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.416826537, 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 511 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.006 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272235+0000", 2024-09-16T04:42:42.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6300000000000004e-07 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272241+0000", 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272255+0000", 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3223e-05 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.272263+0000", 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0199999999999994e-06 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.472829+0000", 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.200566153 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.472837+0000", 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2369999999999992e-06 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.475728+0000", 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028907859999999998 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.475882+0000", 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000154759 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:14.689061+0000", 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.213178642 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:519 3.fs0 3:f1175b03:::smithi16530038-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.686141771999999, 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83548977300000005, 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.011 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 519 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233344+0000", 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233352+0000", 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5100000000000001e-06 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233381+0000", 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8887e-05 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233398+0000", 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7427000000000001e-05 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233732+0000", 2024-09-16T04:42:42.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033334700000000001 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233749+0000", 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7612999999999999e-05 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236234+0000", 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024850919999999999 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:42.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236334+0000", 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100159 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068497+0000", 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83216263000000001 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068656+0000", 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015873499999999999 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068833+0000", 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017668299999999999 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:534 3.fs0 3:f6276c08:::smithi16530038-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.660749264, 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81072121699999999, 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 534 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258737+0000", 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.46e-06 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258746+0000", 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3670000000000003e-06 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258767+0000", 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1253000000000001e-05 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258785+0000", 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8046e-05 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.561788+0000", 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30300253199999999 2024-09-16T04:42:42.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.562084+0000", 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029564400000000002 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.662783+0000", 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10069991 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.663047+0000", 2024-09-16T04:42:42.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026331899999999998 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.069456+0000", 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40640968599999999 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:624 3.fs0 3:f1175b03:::smithi16530038-29:head [copy-from ver 2] snapc 14=[14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:25.140380+0000", 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.779104365999999, 2024-09-16T04:42:42.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.65160655300000003, 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 624 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.140380+0000", 2024-09-16T04:42:42.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.140380+0000", 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.140383+0000", 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.807e-06 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.140395+0000", 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2206e-05 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.140431+0000", 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5373000000000002e-05 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.240514+0000", 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100083308 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.240599+0000", 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4894000000000004e-05 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.240742+0000", 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143079 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.488094+0000", 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.24735194699999999 2024-09-16T04:42:42.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.491566+0000", 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003472461 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.791739+0000", 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.300172997 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:25.791987+0000", 2024-09-16T04:42:42.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024748100000000002 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:713 3.cs0 3:31eec322:::smithi16530038-9:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:29.712823+0000", 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.206661922, 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50665098099999994, 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 713 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.712823+0000", 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.712823+0000", 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.712823+0000", 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6199999999999997e-07 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.712830+0000", 2024-09-16T04:42:42.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3999999999999997e-06 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.712842+0000", 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2717999999999999e-05 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.712852+0000", 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8320000000000001e-06 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.717829+0000", 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049769610000000002 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.718024+0000", 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019467599999999999 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.718856+0000", 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083210999999999995 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.723122+0000", 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004266405 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.219340+0000", 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49621716500000002 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.219474+0000", 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013395200000000001 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.033 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.034 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.034 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:42.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:42:42.036 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:42.031+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:42.136 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:42.135+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:42.163 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:42.163 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:42.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:42:42.237 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:42.231+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:42.300 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:42.300 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:42.300 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:42.300 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:42.300 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:42.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:42:42.362 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:42.359+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:42.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:42.440 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:3 3.5s0 3:a54da884:::smithi16530038-14:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.260881105000003, 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50899456499999995, 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176460+0000", 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176463+0000", 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-16T04:42:42.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176482+0000", 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8975999999999999e-05 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.176519+0000", 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6916999999999999e-05 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.276624+0000", 2024-09-16T04:42:42.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100105124 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478009+0000", 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.201384806 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.478333+0000", 2024-09-16T04:42:42.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000324381 2024-09-16T04:42:42.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.685454+0000", 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20712119400000001 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:361 3.5s0 3:a54da884:::smithi16530038-14:head [read 0~229376,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:42.446 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 44.179472715999999, 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.306317012, 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 361 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257868+0000", 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257869+0000", 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257874+0000", 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.023e-06 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257885+0000", 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-16T04:42:42.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.257892+0000", 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5630000000000003e-06 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.351201+0000", 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093309317000000003 2024-09-16T04:42:42.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.351321+0000", 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119459 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:58.564185+0000", 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21286419300000001 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:392 3.5s0 3:a54da884:::smithi16530038-14:head [copy-from ver 8] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.294938047999999, 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.36993664399999998, 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 392 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.452 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142403+0000", 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142404+0000", 2024-09-16T04:42:42.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.674e-06 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142411+0000", 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0400000000000004e-06 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142438+0000", 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6486e-05 2024-09-16T04:42:42.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142449+0000", 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0835999999999999e-05 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142539+0000", 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9823999999999999e-05 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.142675+0000", 2024-09-16T04:42:42.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013676899999999999 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.294617+0000", 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15194110199999999 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.308789+0000", 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014172736999999999 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.512242+0000", 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20345274799999999 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:00.512339+0000", 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7428000000000003e-05 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:520 3.5s0 3:a54c6f35:::smithi16530038-38:head [append 0~0] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.232996+0000", 2024-09-16T04:42:42.457 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.20434513, 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41309273600000002, 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 520 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.232996+0000", 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.232996+0000", 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.232999+0000", 2024-09-16T04:42:42.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8739999999999999e-06 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233010+0000", 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.099e-05 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233044+0000", 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4391999999999999e-05 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233067+0000", 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3323e-05 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233303+0000", 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023559899999999999 2024-09-16T04:42:42.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233625+0000", 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032232700000000002 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236680+0000", 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003055168 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545473+0000", 2024-09-16T04:42:42.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30879239800000002 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.645946+0000", 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100472882 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646088+0000", 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142783 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:521 3.5s0 3:a54c6f35:::smithi16530038-38:head [rollback f] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.204217293999999, 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.514017481, 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.464 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.465 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.465 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 521 2024-09-16T04:42:42.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.465 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233125+0000", 2024-09-16T04:42:42.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.477e-06 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233131+0000", 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9660000000000001e-06 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233176+0000", 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5086000000000002e-05 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233194+0000", 2024-09-16T04:42:42.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7706999999999998e-05 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.239157+0000", 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059635820000000003 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.239414+0000", 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025656799999999999 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.240993+0000", 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015795749999999999 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.545483+0000", 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.304489921 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746960+0000", 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.201476977 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747141+0000", 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018062199999999999 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:522 3.5s0 3:a54c6f35:::smithi16530038-38:head [append 0~0] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.233288+0000", 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.204053311999999, 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51453239399999995, 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.470 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 522 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233288+0000", 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233288+0000", 2024-09-16T04:42:42.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233289+0000", 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4169999999999999e-06 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233296+0000", 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3830000000000003e-06 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233318+0000", 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.145e-05 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233334+0000", 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6429000000000001e-05 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.342490+0000", 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10915607099999999 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.342926+0000", 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043572400000000002 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.344296+0000", 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001369566 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.645708+0000", 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30141223700000003 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747663+0000", 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10195557800000001 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747820+0000", 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156539 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:5 3.5s0 3:a2d95112:::smithi16530038-40:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.237468+0000", 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.199872863, 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71584401900000005, 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237468+0000", 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237468+0000", 2024-09-16T04:42:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237471+0000", 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9440000000000001e-06 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237487+0000", 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5722999999999998e-05 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237532+0000", 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5116e-05 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.237554+0000", 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1766000000000001e-05 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.850151+0000", 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61259759800000002 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.850307+0000", 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015556599999999999 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.953312+0000", 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103005306 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.481 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=33, last_complete=65'23, committed=1, applied=1))", 2024-09-16T04:42:42.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.243164+0000", 2024-09-16T04:42:42.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.194176511999999, 2024-09-16T04:42:42.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40294550099999998, 2024-09-16T04:42:42.482 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.482 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.482 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243164+0000", 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243164+0000", 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243166+0000", 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.84e-06 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243176+0000", 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0152999999999999e-05 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243277+0000", 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101006 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243293+0000", 2024-09-16T04:42:42.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5896000000000002e-05 2024-09-16T04:42:42.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.645804+0000", 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402510545 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646110+0000", 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000306061 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=34, last_complete=65'24, committed=1, applied=1))", 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.243343+0000", 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.193997837000001, 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40290150200000002, 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243343+0000", 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243343+0000", 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243344+0000", 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.345e-06 2024-09-16T04:42:42.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243353+0000", 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.243376+0000", 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3419000000000001e-05 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.343476+0000", 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100099434 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646210+0000", 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30273381399999999 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.646244+0000", 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4866999999999997e-05 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.490 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=34, last_complete=65'24, committed=1, applied=1))", 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.247549+0000", 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.189791692, 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49961050899999998, 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.491 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247549+0000", 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247549+0000", 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247551+0000", 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.95e-06 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247563+0000", 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2300000000000001e-05 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247590+0000", 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7016999999999999e-05 2024-09-16T04:42:42.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.247611+0000", 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0696e-05 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746819+0000", 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49920759799999997 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747160+0000", 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034094799999999999 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=34, last_complete=65'24, committed=1, applied=1))", 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.250731+0000", 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.186609785000002, 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49578053700000002, 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.495 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250731+0000", 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250731+0000", 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250733+0000", 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7689999999999999e-06 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250741+0000", 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5939999999999994e-06 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250766+0000", 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4206000000000002e-05 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.250780+0000", 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3963e-05 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746474+0000", 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49569433200000002 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.746512+0000", 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7673000000000003e-05 2024-09-16T04:42:42.498 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.346174+0000", 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.091167178999999, 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40273614899999999, 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346174+0000", 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346174+0000", 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346175+0000", 2024-09-16T04:42:42.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.517e-06 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346184+0000", 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346208+0000", 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4063e-05 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346228+0000", 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9148999999999999e-05 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747539+0000", 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40131134099999999 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.748910+0000", 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001370806 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.346887+0000", 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.090453797999999, 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40035135100000002, 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346887+0000", 2024-09-16T04:42:42.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346887+0000", 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346889+0000", 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8700000000000001e-06 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346898+0000", 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4360000000000008e-06 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346928+0000", 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9904e-05 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.346950+0000", 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1460000000000001e-05 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747213+0000", 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40026329399999999 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747238+0000", 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5386999999999999e-05 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.347600+0000", 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.089740997, 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39977881599999998, 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347600+0000", 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347600+0000", 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347602+0000", 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.86e-06 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347612+0000", 2024-09-16T04:42:42.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0530000000000001e-05 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347643+0000", 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1066999999999999e-05 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.347664+0000", 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.048e-05 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747346+0000", 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39968240500000002 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747379+0000", 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2474000000000001e-05 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 65/37 ECSubWriteReply(tid=35, last_complete=65'25, committed=1, applied=1))", 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.434934+0000", 2024-09-16T04:42:42.510 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.002406563000001, 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31236023899999998, 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434934+0000", 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434934+0000", 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434936+0000", 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-06 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434946+0000", 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0723999999999999e-05 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434978+0000", 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1933000000000001e-05 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.434999+0000", 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0109999999999999e-05 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747273+0000", 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31227403300000001 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.747295+0000", 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1946e-05 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:653 3.5s0 3:a54da884:::smithi16530038-14:head [rollback 13] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:26.182867+0000", 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.254473361999999, 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.507240837, 2024-09-16T04:42:42.514 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 653 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.182867+0000", 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.182867+0000", 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.182868+0000", 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9700000000000005e-07 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.182889+0000", 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0653e-05 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.182904+0000", 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-05 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.282997+0000", 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10009259399999999 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283196+0000", 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000199798 2024-09-16T04:42:42.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283357+0000", 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160996 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.284274+0000", 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00091622299999999997 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.487678+0000", 2024-09-16T04:42:42.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20340476099999999 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.689904+0000", 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20222573999999999 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.690108+0000", 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020424500000000001 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:654 3.5s0 3:a54da884:::smithi16530038-14:head [append 0~0] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:26.283035+0000", 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.154305678, 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.408127458, 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 654 2024-09-16T04:42:42.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283035+0000", 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283035+0000", 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283037+0000", 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8759999999999999e-06 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283046+0000", 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283069+0000", 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3200000000000001e-05 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.283082+0000", 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3200000000000001e-05 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.385138+0000", 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10205558300000001 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.385455+0000", 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000317547 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.386676+0000", 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001220311 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.487684+0000", 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101007984 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.690957+0000", 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20327341900000001 2024-09-16T04:42:42.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.691163+0000", 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020546799999999999 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.1ds1 76/75 ECSubRead(tid=53, to_read={3:bc30766e:::smithi16530038-11:head=0,430080,0}, subchunks={3:bc30766e:::smithi16530038-11:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:26.493808+0000", 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.943533236, 2024-09-16T04:42:42.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49964349800000002, 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.493808+0000", 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.493808+0000", 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.493809+0000", 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8169999999999999e-06 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.493828+0000", 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8346e-05 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.493868+0000", 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0086999999999998e-05 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.593967+0000", 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10009915699999999 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.993221+0000", 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39925355899999998 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.993451+0000", 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023053200000000001 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(3.1ds1 epoch 76/75 rep_tid 54 entries 76'113 (0'0) error 3:bc0e3a40:::smithi16530038-18:head by client.4728.0:655 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:26.594020+0000", 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.843320856, 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50072432200000005, 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.529 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.532 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.532 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594020+0000", 2024-09-16T04:42:42.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594020+0000", 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594021+0000", 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0499999999999999e-06 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594036+0000", 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4956999999999999e-05 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594071+0000", 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5469000000000002e-05 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594097+0000", 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5380000000000001e-05 2024-09-16T04:42:42.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:27.093689+0000", 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49959195499999998 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:27.094744+0000", 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010555110000000001 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.1ds1 76/75 ECSubWrite(tid=55, reqid=osd.5.0:55, at_version=76'115, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:26.594120+0000", 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.843221206999999, 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50135917200000002, 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594120+0000", 2024-09-16T04:42:42.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594120+0000", 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594120+0000", 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.37e-07 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:42.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594151+0000", 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0973000000000003e-05 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594168+0000", 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6776999999999999e-05 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:26.594181+0000", 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3213e-05 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:27.094075+0000", 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49989436900000001 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:27.094093+0000", 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7147e-05 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:27.095408+0000", 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001314976 2024-09-16T04:42:42.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:27.095479+0000", 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000006e-05 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:42.540 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:42.541 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:42.451+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:42.543 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:42:42.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:329: done (3 left) 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:330: read oid 35 snap -1 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:330: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:331: read oid 23 snap -1 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:331: expect (ObjNum 111 snap 20 seq_num 111) 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:332: copy_from oid 7 from oid 21 current snap is 26 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:324: finishing write tid 5 to smithi16530038-45 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:324: finishing write tid 6 to smithi16530038-45 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:324: oid 45 updating version 0 to 55 2024-09-16T04:42:42.544 INFO:tasks.rados.rados.0.smithi165.stdout:324: oid 45 updating version 55 to 58 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:324: oid 45 version 58 is already newer than 57 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:324: oid 45 updating version 58 to 61 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:324: oid 45 version 61 is already newer than 56 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 61 (ObjNum 123 snap 25 seq_num 123) dirty exists 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:324: left oid 45 (ObjNum 123 snap 25 seq_num 123) 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:324: done (5 left) 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:333: delete oid 31 current snap is 26 2024-09-16T04:42:42.545 INFO:tasks.rados.rados.0.smithi165.stdout:328: expect (ObjNum 121 snap 24 seq_num 121) 2024-09-16T04:42:42.545 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:42.539+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:42.572 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:42.593 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:42.593 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:42.642 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:42.642 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:42.643 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:42.639+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:42.643 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 1 to smithi16530038-43 2024-09-16T04:42:42.643 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 2 to smithi16530038-43 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 3 to smithi16530038-43 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 4 to smithi16530038-43 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 5 to smithi16530038-43 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 6 to smithi16530038-43 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 43 updating version 0 to 22 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 43 updating version 22 to 23 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 43 version 23 is already newer than 19 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 43 version 23 is already newer than 21 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 43 version 23 is already newer than 20 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 23 (ObjNum 124 snap 26 seq_num 124) dirty exists 2024-09-16T04:42:42.644 INFO:tasks.rados.rados.0.smithi165.stdout:327: left oid 43 (ObjNum 124 snap 26 seq_num 124) 2024-09-16T04:42:42.676 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:42.677 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:42.702 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:42.702 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:42.719 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:42.743 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:42.739+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:42.831 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:42.832 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:42.844 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:42.839+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:42.887 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:42.887 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:42.903 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:42.929 INFO:tasks.rados.rados.0.smithi165.stdout:327: done (5 left) 2024-09-16T04:42:42.929 INFO:tasks.rados.rados.0.smithi165.stdout:328: done (4 left) 2024-09-16T04:42:42.929 INFO:tasks.rados.rados.0.smithi165.stdout:333: done (3 left) 2024-09-16T04:42:42.929 INFO:tasks.rados.rados.0.smithi165.stdout:334: snap_create 2024-09-16T04:42:42.946 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:42.939+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:42.949 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:42.949 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:43.032 INFO:tasks.rados.rados.0.smithi165.stdout:331: expect (ObjNum 111 snap 20 seq_num 111) 2024-09-16T04:42:43.037 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:43.038 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:43.046 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:43.043+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:43.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:43.059 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:43.095 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:43.096 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:43.130 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:43.131 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:43.146 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:43.143+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:43.154 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:43.154 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:43.180 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:43.180 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:43.213 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:43.219 INFO:tasks.rados.rados.0.smithi165.stdout:330: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T04:42:43.247 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:43.243+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:43.300 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:43.301 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:43.323 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:43.324 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:43.348 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:43.343+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:43.348 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:43.349 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:43.373 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:43.379 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:43.379 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:43.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:43.402 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:43.402 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:43.448 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:43.443+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:43.509 INFO:tasks.rados.rados.0.smithi165.stdout:330: done (3 left) 2024-09-16T04:42:43.509 INFO:tasks.rados.rados.0.smithi165.stdout:331: done (2 left) 2024-09-16T04:42:43.509 INFO:tasks.rados.rados.0.smithi165.stdout:334: done (1 left) 2024-09-16T04:42:43.509 INFO:tasks.rados.rados.0.smithi165.stdout:335: delete oid 23 current snap is 27 2024-09-16T04:42:43.512 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:43.512 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:43.527 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:43.538 INFO:tasks.rados.rados.0.smithi165.stdout:332: finishing copy_from to smithi16530038-7 2024-09-16T04:42:43.538 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 19 (ObjNum 94 snap 16 seq_num 94) dirty exists 2024-09-16T04:42:43.542 INFO:tasks.rados.rados.0.smithi165.stdout:332: finishing copy_from racing read to smithi16530038-7 2024-09-16T04:42:43.549 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:43.547+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:43.648 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:43.649 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:43.650 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:43.647+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:43.664 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:42:43.751 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:43.747+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:43.791 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:43.792 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:43.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:42:43.817 INFO:tasks.rados.rados.0.smithi165.stdout:332: done (1 left) 2024-09-16T04:42:43.818 INFO:tasks.rados.rados.0.smithi165.stdout:335: done (0 left) 2024-09-16T04:42:43.818 INFO:tasks.rados.rados.0.smithi165.stdout:336: delete oid 8 current snap is 27 2024-09-16T04:42:43.852 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:43.847+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:43.953 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:43.947+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:43.974 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:43.975 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:43.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:42:44.020 INFO:tasks.rados.rados.0.smithi165.stdout:336: done (0 left) 2024-09-16T04:42:44.021 INFO:tasks.rados.rados.0.smithi165.stdout:337: delete oid 49 current snap is 27 2024-09-16T04:42:44.037 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:42:44.037 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:42:44.038 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-16T04:42:44.038 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 16 2024-09-16T04:42:44.038 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:42:44.055 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:44.051+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:44.120 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:44.120 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:44.135 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:42:44.156 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:44.155+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:44.224 INFO:tasks.rados.rados.0.smithi165.stdout:337: done (0 left) 2024-09-16T04:42:44.224 INFO:tasks.rados.rados.0.smithi165.stdout:338: copy_from oid 33 from oid 9 current snap is 27 2024-09-16T04:42:44.225 INFO:tasks.rados.rados.0.smithi165.stdout:339: read oid 27 snap -1 2024-09-16T04:42:44.225 INFO:tasks.rados.rados.0.smithi165.stdout:339: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:44.225 INFO:tasks.rados.rados.0.smithi165.stdout:340: rollback oid 42 current snap is 27 2024-09-16T04:42:44.225 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 24 2024-09-16T04:42:44.225 INFO:tasks.rados.rados.0.smithi165.stdout:341: read oid 29 snap -1 2024-09-16T04:42:44.225 INFO:tasks.rados.rados.0.smithi165.stdout:341: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:44.225 INFO:tasks.rados.rados.0.smithi165.stdout:342: delete oid 19 current snap is 27 2024-09-16T04:42:44.256 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:44.257 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:44.257 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:44.251+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:44.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:42:44.357 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:42:44.358 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:42:44.359 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:44.351+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:44.381 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-16T04:42:43.507725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89932339,"num_objects":100,"num_object_clones":50,"num_object_copies":680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":49,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":7,"num_read":247,"num_read_kb":74863,"num_write":369,"num_write_kb":29958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4505600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1652,"ondisk_log_size":1652,"up":270,"acting":270,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":277,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":679520,"kb_used_data":180492,"kb_used_omap":24,"kb_used_meta":498663,"kb_avail":1509269920,"statfs":{"total":1546188226560,"available":1545492398080,"internally_reserved":0,"allocated":184823808,"data_stored":242873941,"data_compressed":3245739,"data_compressed_allocated":4460544,"data_compressed_original":9781248,"omap_allocated":25522,"internal_metadata":510630990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1097728,"num_objects":-7,"num_object_clones":-4,"num_object_copies":-49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":1,"num_read":-14,"num_read_kb":8717,"num_write":-37,"num_write_kb":-8816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11,"num_bytes_recovered":-3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038590"},"pg_stats":[{"pgid":"3.23","version":"60'17","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.684701+0000","last_change":"2024-09-16T04:42:38.574965+0000","last_active":"2024-09-16T04:42:39.684701+0000","last_peered":"2024-09-16T04:42:39.684701+0000","last_clean":"2024-09-16T04:42:39.684701+0000","last_became_active":"2024-09-16T04:42:20.647183+0000","last_became_peered":"2024-09-16T04:42:20.647183+0000","last_unstale":"2024-09-16T04:42:39.684701+0000","last_undegraded":"2024-09-16T04:42:39.684701+0000","last_fullsized":"2024-09-16T04:42:39.684701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'17","last_scrub_stamp":"2024-09-16T04:42:25.580862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.580862+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:47.571529+0000","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0.00082925399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,0,15,12,1],"acting":[8,4,11,0,15,12,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.22","version":"72'17","reported_seq":140,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.471038+0000","last_change":"2024-09-16T04:42:38.779262+0000","last_active":"2024-09-16T04:42:39.471038+0000","last_peered":"2024-09-16T04:42:39.471038+0000","last_clean":"2024-09-16T04:42:39.471038+0000","last_became_active":"2024-09-16T04:42:20.464635+0000","last_became_peered":"2024-09-16T04:42:20.464635+0000","last_unstale":"2024-09-16T04:42:39.471038+0000","last_undegraded":"2024-09-16T04:42:39.471038+0000","last_fullsized":"2024-09-16T04:42:39.471038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":6,"last_scrub":"72'17","last_scrub_stamp":"2024-09-16T04:42:26.786330+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.786330+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.851433+0000","scrub_duration":843,"objects_trimmed":0,"snaptrim_duration":0.00133172,"stat_sum":{"num_bytes":2048000,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,11,4,7,14],"acting":[2,5,0,11,4,7,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.21","version":"82'37","reported_seq":145,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.471028+0000","last_change":"2024-09-16T04:42:38.678303+0000","last_active":"2024-09-16T04:42:39.471028+0000","last_peered":"2024-09-16T04:42:39.471028+0000","last_clean":"2024-09-16T04:42:39.471028+0000","last_became_active":"2024-09-16T04:42:20.469617+0000","last_became_peered":"2024-09-16T04:42:20.469617+0000","last_unstale":"2024-09-16T04:42:39.471028+0000","last_undegraded":"2024-09-16T04:42:39.471028+0000","last_fullsized":"2024-09-16T04:42:39.471028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'31","last_scrub_stamp":"2024-09-16T04:42:26.688435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.688435+0000","objects_scrubbed":1,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:52.622386+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0.00064411299999999998,"stat_sum":{"num_bytes":311296,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,5,15,4,0,12],"acting":[2,10,5,15,4,0,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.20","version":"36'4","reported_seq":116,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.572998+0000","last_change":"2024-09-16T04:42:38.673560+0000","last_active":"2024-09-16T04:42:38.572998+0000","last_peered":"2024-09-16T04:42:38.572998+0000","last_clean":"2024-09-16T04:42:38.572998+0000","last_became_active":"2024-09-16T04:42:20.360418+0000","last_became_peered":"2024-09-16T04:42:20.360418+0000","last_unstale":"2024-09-16T04:42:38.572998+0000","last_undegraded":"2024-09-16T04:42:38.572998+0000","last_fullsized":"2024-09-16T04:42:38.572998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":6,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.091321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.091321+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:48.896941+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0.100854454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,13,5,3,14,15],"acting":[6,2,13,5,3,14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.1f","version":"79'56","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.589258+0000","last_change":"2024-09-16T04:42:40.691028+0000","last_active":"2024-09-16T04:42:40.589258+0000","last_peered":"2024-09-16T04:42:40.589258+0000","last_clean":"2024-09-16T04:42:40.589258+0000","last_became_active":"2024-09-16T04:42:20.460902+0000","last_became_peered":"2024-09-16T04:42:20.460902+0000","last_unstale":"2024-09-16T04:42:40.589258+0000","last_undegraded":"2024-09-16T04:42:40.589258+0000","last_fullsized":"2024-09-16T04:42:40.589258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'52","last_scrub_stamp":"2024-09-16T04:42:25.895250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.895250+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:36.087614+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0.102361334,"stat_sum":{"num_bytes":1867776,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5474,"num_write":4,"num_write_kb":1824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,15,8,0,2,11,14],"acting":[1,15,8,0,2,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"8"},{"start":"12","length":"6"}]},{"pgid":"3.1e","version":"82'67","reported_seq":186,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.584967+0000","last_change":"2024-09-16T04:42:38.476249+0000","last_active":"2024-09-16T04:42:39.584967+0000","last_peered":"2024-09-16T04:42:39.584967+0000","last_clean":"2024-09-16T04:42:39.584967+0000","last_became_active":"2024-09-16T04:42:20.562288+0000","last_became_peered":"2024-09-16T04:42:20.562288+0000","last_unstale":"2024-09-16T04:42:39.584967+0000","last_undegraded":"2024-09-16T04:42:39.584967+0000","last_fullsized":"2024-09-16T04:42:39.584967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"74'53","last_scrub_stamp":"2024-09-16T04:42:27.214170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.214170+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.876918+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0.00096192999999999995,"stat_sum":{"num_bytes":3047424,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":8930,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,5,1,14,3,10],"acting":[8,6,5,1,14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.1d","version":"88'139","reported_seq":246,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.614457+0000","last_change":"2024-09-16T04:42:39.513068+0000","last_active":"2024-09-16T04:42:39.614457+0000","last_peered":"2024-09-16T04:42:39.614457+0000","last_clean":"2024-09-16T04:42:39.614457+0000","last_became_active":"2024-09-16T04:42:26.481390+0000","last_became_peered":"2024-09-16T04:42:26.481390+0000","last_unstale":"2024-09-16T04:42:39.614457+0000","last_undegraded":"2024-09-16T04:42:39.614457+0000","last_fullsized":"2024-09-16T04:42:39.614457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'133","last_scrub_stamp":"2024-09-16T04:42:36.357029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:36.357029+0000","objects_scrubbed":6,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:40.528940+0000","scrub_duration":979,"objects_trimmed":1,"snaptrim_duration":1.0174909679999999,"stat_sum":{"num_bytes":8110080,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4897,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,11,9,14,13,3],"acting":[5,15,11,9,14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.1c","version":"90'58","reported_seq":388,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:42:42.926352+0000","last_change":"2024-09-16T04:42:43.029984+0000","last_active":"2024-09-16T04:42:42.926352+0000","last_peered":"2024-09-16T04:42:42.926352+0000","last_clean":"2024-09-16T04:42:42.926352+0000","last_became_active":"2024-09-16T04:42:20.367551+0000","last_became_peered":"2024-09-16T04:42:20.367551+0000","last_unstale":"2024-09-16T04:42:42.926352+0000","last_undegraded":"2024-09-16T04:42:42.926352+0000","last_fullsized":"2024-09-16T04:42:42.926352+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'54","last_scrub_stamp":"2024-09-16T04:42:25.992723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.992723+0000","objects_scrubbed":2,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.338346+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.41334522200000001,"stat_sum":{"num_bytes":229376,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":673,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,4,13,5,12,9],"acting":[11,7,4,13,5,12,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"f"}]},{"pgid":"3.1b","version":"63'44","reported_seq":116,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.785244+0000","last_change":"2024-09-16T04:42:38.785907+0000","last_active":"2024-09-16T04:42:38.785244+0000","last_peered":"2024-09-16T04:42:38.785244+0000","last_clean":"2024-09-16T04:42:38.785244+0000","last_became_active":"2024-09-16T04:42:20.564769+0000","last_became_peered":"2024-09-16T04:42:20.564769+0000","last_unstale":"2024-09-16T04:42:38.785244+0000","last_undegraded":"2024-09-16T04:42:38.785244+0000","last_fullsized":"2024-09-16T04:42:38.785244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'44","last_scrub_stamp":"2024-09-16T04:42:25.681915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.681915+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:43.179979+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0.001000941,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,5,8,12,13,7],"acting":[10,15,5,8,12,13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.1a","version":"87'53","reported_seq":161,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.469154+0000","last_change":"2024-09-16T04:42:38.569097+0000","last_active":"2024-09-16T04:42:38.469154+0000","last_peered":"2024-09-16T04:42:38.469154+0000","last_clean":"2024-09-16T04:42:38.469154+0000","last_became_active":"2024-09-16T04:42:20.461907+0000","last_became_peered":"2024-09-16T04:42:20.461907+0000","last_unstale":"2024-09-16T04:42:38.469154+0000","last_undegraded":"2024-09-16T04:42:38.469154+0000","last_fullsized":"2024-09-16T04:42:38.469154+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'51","last_scrub_stamp":"2024-09-16T04:42:35.060495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.060495+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:54.359905+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0.10027604800000001,"stat_sum":{"num_bytes":1916928,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,12,10,13,9,7,11],"acting":[14,12,10,13,9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.19","version":"87'40","reported_seq":303,"reported_epoch":88,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T04:42:38.582084+0000","last_change":"2024-09-16T04:42:09.757868+0000","last_active":"2024-09-16T04:42:38.582084+0000","last_peered":"2024-09-16T04:42:38.582084+0000","last_clean":"2024-09-16T04:42:06.828625+0000","last_became_active":"2024-09-16T04:42:09.651531+0000","last_became_peered":"2024-09-16T04:42:09.651531+0000","last_unstale":"2024-09-16T04:42:38.582084+0000","last_undegraded":"2024-09-16T04:42:38.582084+0000","last_fullsized":"2024-09-16T04:42:38.582084+0000","mapping_epoch":58,"log_start":"44'22","ondisk_log_start":"44'22","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":18,"log_dups_size":22,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.098972921000000005,"stat_sum":{"num_bytes":2637824,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":49,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":10899,"num_write":7,"num_write_kb":672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[14,1,0,3,15,8,10],"avail_no_missing":["14(0)","0(2)","1(1)","3(3)","8(5)","10(6)","15(4)"],"object_location_counts":[{"shards":"0(2),1(1),3(3),8(5),10(6),14(0),15(4)","objects":7}],"blocked_by":[],"up_primary":8,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.18","version":"44'8","reported_seq":346,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.474871+0000","last_change":"2024-09-16T04:42:40.476323+0000","last_active":"2024-09-16T04:42:40.474871+0000","last_peered":"2024-09-16T04:42:40.474871+0000","last_clean":"2024-09-16T04:42:40.474871+0000","last_became_active":"2024-09-16T04:42:12.629523+0000","last_became_peered":"2024-09-16T04:42:12.629523+0000","last_unstale":"2024-09-16T04:42:40.474871+0000","last_undegraded":"2024-09-16T04:42:40.474871+0000","last_fullsized":"2024-09-16T04:42:40.474871+0000","mapping_epoch":61,"log_start":"44'8","ondisk_log_start":"44'8","created":55,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'8","last_scrub_stamp":"2024-09-16T04:42:14.378029+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:14.378029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:39.473287+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0.0020484140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,0,4,5,7,12],"acting":[13,6,0,4,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"8"},{"start":"12","length":"6"}]},{"pgid":"3.17","version":"44'6","reported_seq":216,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.514171+0000","last_change":"2024-09-16T04:42:38.595930+0000","last_active":"2024-09-16T04:42:39.514171+0000","last_peered":"2024-09-16T04:42:39.514171+0000","last_clean":"2024-09-16T04:42:39.514171+0000","last_became_active":"2024-09-16T04:42:04.604172+0000","last_became_peered":"2024-09-16T04:42:04.604172+0000","last_unstale":"2024-09-16T04:42:39.514171+0000","last_undegraded":"2024-09-16T04:42:39.514171+0000","last_fullsized":"2024-09-16T04:42:39.514171+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'6","last_scrub_stamp":"2024-09-16T04:42:11.540212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:11.540212+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:23.543926+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0.100357654,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3554,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.16","version":"92'16","reported_seq":224,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:42:42.823223+0000","last_change":"2024-09-16T04:42:42.825193+0000","last_active":"2024-09-16T04:42:42.823223+0000","last_peered":"2024-09-16T04:42:42.823223+0000","last_clean":"2024-09-16T04:42:42.823223+0000","last_became_active":"2024-09-16T04:42:04.405792+0000","last_became_peered":"2024-09-16T04:42:04.405792+0000","last_unstale":"2024-09-16T04:42:42.823223+0000","last_undegraded":"2024-09-16T04:42:42.823223+0000","last_fullsized":"2024-09-16T04:42:42.823223+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:57.977558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:57.977558+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:09.262995+0000","scrub_duration":515,"objects_trimmed":2,"snaptrim_duration":0.308440145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,4,8,9,5,10],"acting":[11,15,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"f"}]},{"pgid":"3.15","version":"36'6","reported_seq":226,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.588243+0000","last_change":"2024-09-16T04:42:40.592315+0000","last_active":"2024-09-16T04:42:40.588243+0000","last_peered":"2024-09-16T04:42:40.588243+0000","last_clean":"2024-09-16T04:42:40.588243+0000","last_became_active":"2024-09-16T04:42:05.419352+0000","last_became_peered":"2024-09-16T04:42:05.419352+0000","last_unstale":"2024-09-16T04:42:40.588243+0000","last_undegraded":"2024-09-16T04:42:40.588243+0000","last_fullsized":"2024-09-16T04:42:40.588243+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'6","last_scrub_stamp":"2024-09-16T04:42:10.772708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.772708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:30.781774+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.0049613000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,15],"acting":[9,6,3,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"8"},{"start":"12","length":"6"}]},{"pgid":"3.14","version":"88'81","reported_seq":279,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.472106+0000","last_change":"2024-09-16T04:42:38.675193+0000","last_active":"2024-09-16T04:42:39.472106+0000","last_peered":"2024-09-16T04:42:39.472106+0000","last_clean":"2024-09-16T04:42:39.472106+0000","last_became_active":"2024-09-16T04:42:04.626975+0000","last_became_peered":"2024-09-16T04:42:04.626975+0000","last_unstale":"2024-09-16T04:42:39.472106+0000","last_undegraded":"2024-09-16T04:42:39.472106+0000","last_fullsized":"2024-09-16T04:42:39.472106+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:00.510344+0000","scrub_duration":899,"objects_trimmed":0,"snaptrim_duration":0.20664179899999999,"stat_sum":{"num_bytes":8290304,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":8,"num_read_kb":5331,"num_write":16,"num_write_kb":5072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4423680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,15,1,13,8,4],"acting":[3,9,15,1,13,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.476126+0000","last_change":"2024-09-16T04:41:30.108187+0000","last_active":"2024-09-16T04:42:38.476126+0000","last_peered":"2024-09-16T04:42:38.476126+0000","last_clean":"2024-09-16T04:42:38.476126+0000","last_became_active":"2024-09-16T04:41:27.699143+0000","last_became_peered":"2024-09-16T04:41:27.699143+0000","last_unstale":"2024-09-16T04:42:38.476126+0000","last_undegraded":"2024-09-16T04:42:38.476126+0000","last_fullsized":"2024-09-16T04:42:38.476126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:39.355644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":293,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:42:42.507476+0000","last_change":"2024-09-16T04:42:42.508726+0000","last_active":"2024-09-16T04:42:42.507476+0000","last_peered":"2024-09-16T04:42:42.507476+0000","last_clean":"2024-09-16T04:42:42.507476+0000","last_became_active":"2024-09-16T04:41:48.483004+0000","last_became_peered":"2024-09-16T04:41:48.483004+0000","last_unstale":"2024-09-16T04:42:42.507476+0000","last_undegraded":"2024-09-16T04:42:42.507476+0000","last_fullsized":"2024-09-16T04:42:42.507476+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:53.522506+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.002213633,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"f"}]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.585176+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:42:39.585176+0000","last_peered":"2024-09-16T04:42:39.585176+0000","last_clean":"2024-09-16T04:42:39.585176+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:42:39.585176+0000","last_undegraded":"2024-09-16T04:42:39.585176+0000","last_fullsized":"2024-09-16T04:42:39.585176+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:48.225715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":276,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.473117+0000","last_change":"2024-09-16T04:42:38.474083+0000","last_active":"2024-09-16T04:42:38.473117+0000","last_peered":"2024-09-16T04:42:38.473117+0000","last_clean":"2024-09-16T04:42:38.473117+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:42:38.473117+0000","last_undegraded":"2024-09-16T04:42:38.473117+0000","last_fullsized":"2024-09-16T04:42:38.473117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:42:00.916431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.916431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:06.240257+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.001348938,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.672646+0000","last_change":"2024-09-16T04:41:30.210057+0000","last_active":"2024-09-16T04:42:38.672646+0000","last_peered":"2024-09-16T04:42:38.672646+0000","last_clean":"2024-09-16T04:42:38.672646+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:42:38.672646+0000","last_undegraded":"2024-09-16T04:42:38.672646+0000","last_fullsized":"2024-09-16T04:42:38.672646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:40.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"36'10","reported_seq":298,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.584892+0000","last_change":"2024-09-16T04:42:38.476315+0000","last_active":"2024-09-16T04:42:39.584892+0000","last_peered":"2024-09-16T04:42:39.584892+0000","last_clean":"2024-09-16T04:42:39.584892+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:42:39.584892+0000","last_undegraded":"2024-09-16T04:42:39.584892+0000","last_fullsized":"2024-09-16T04:42:39.584892+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:59.997918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.997918+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:09.228828+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.001180323,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"2.4","version":"0'0","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.473201+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:42:38.473201+0000","last_peered":"2024-09-16T04:42:38.473201+0000","last_clean":"2024-09-16T04:42:38.473201+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:42:38.473201+0000","last_undegraded":"2024-09-16T04:42:38.473201+0000","last_fullsized":"2024-09-16T04:42:38.473201+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"88'70","reported_seq":347,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.203878+0000","last_change":"2024-09-16T04:42:39.206295+0000","last_active":"2024-09-16T04:42:39.203878+0000","last_peered":"2024-09-16T04:42:39.203878+0000","last_clean":"2024-09-16T04:42:39.203878+0000","last_became_active":"2024-09-16T04:41:48.475648+0000","last_became_peered":"2024-09-16T04:41:48.475648+0000","last_unstale":"2024-09-16T04:42:39.203878+0000","last_undegraded":"2024-09-16T04:42:39.203878+0000","last_fullsized":"2024-09-16T04:42:39.203878+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:07.004454+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.61592117499999999,"stat_sum":{"num_bytes":7716864,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":9,"num_read_kb":676,"num_write":36,"num_write_kb":4208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.3","version":"90'39","reported_seq":340,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.628334+0000","last_change":"2024-09-16T04:42:40.631657+0000","last_active":"2024-09-16T04:42:40.628334+0000","last_peered":"2024-09-16T04:42:40.628334+0000","last_clean":"2024-09-16T04:42:40.628334+0000","last_became_active":"2024-09-16T04:42:18.590615+0000","last_became_peered":"2024-09-16T04:42:18.590615+0000","last_unstale":"2024-09-16T04:42:40.628334+0000","last_undegraded":"2024-09-16T04:42:40.628334+0000","last_fullsized":"2024-09-16T04:42:40.628334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'21","last_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_deep_scrub":"72'21","last_deep_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.317975+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:37.039156+0000","scrub_duration":643,"objects_trimmed":1,"snaptrim_duration":0.156617331,"stat_sum":{"num_bytes":4456448,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"8"},{"start":"12","length":"6"}]},{"pgid":"2.2","version":"28'2","reported_seq":143,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.471867+0000","last_change":"2024-09-16T04:41:29.619423+0000","last_active":"2024-09-16T04:42:39.471867+0000","last_peered":"2024-09-16T04:42:39.471867+0000","last_clean":"2024-09-16T04:42:39.471867+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:42:39.471867+0000","last_undegraded":"2024-09-16T04:42:39.471867+0000","last_fullsized":"2024-09-16T04:42:39.471867+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.779838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":295,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.468984+0000","last_change":"2024-09-16T04:42:38.469880+0000","last_active":"2024-09-16T04:42:38.468984+0000","last_peered":"2024-09-16T04:42:38.468984+0000","last_clean":"2024-09-16T04:42:38.468984+0000","last_became_active":"2024-09-16T04:42:18.592746+0000","last_became_peered":"2024-09-16T04:42:18.592746+0000","last_unstale":"2024-09-16T04:42:38.468984+0000","last_undegraded":"2024-09-16T04:42:38.468984+0000","last_fullsized":"2024-09-16T04:42:38.468984+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.089771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.089771+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:41.591886+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.001424979,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"2.1","version":"0'0","reported_seq":142,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.474317+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:42:40.474317+0000","last_peered":"2024-09-16T04:42:40.474317+0000","last_clean":"2024-09-16T04:42:40.474317+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:42:40.474317+0000","last_undegraded":"2024-09-16T04:42:40.474317+0000","last_fullsized":"2024-09-16T04:42:40.474317+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:42.668549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"82'47","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.473001+0000","last_change":"2024-09-16T04:42:38.473968+0000","last_active":"2024-09-16T04:42:38.473001+0000","last_peered":"2024-09-16T04:42:38.473001+0000","last_clean":"2024-09-16T04:42:38.473001+0000","last_became_active":"2024-09-16T04:42:18.991363+0000","last_became_peered":"2024-09-16T04:42:18.991363+0000","last_unstale":"2024-09-16T04:42:38.473001+0000","last_undegraded":"2024-09-16T04:42:38.473001+0000","last_fullsized":"2024-09-16T04:42:38.473001+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'35","last_scrub_stamp":"2024-09-16T04:42:24.144068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.144068+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:41.514001+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0.001276386,"stat_sum":{"num_bytes":2637824,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.471744+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:42:39.471744+0000","last_peered":"2024-09-16T04:42:39.471744+0000","last_clean":"2024-09-16T04:42:39.471744+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:42:39.471744+0000","last_undegraded":"2024-09-16T04:42:39.471744+0000","last_fullsized":"2024-09-16T04:42:39.471744+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:55.372936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"59'15","reported_seq":315,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.575107+0000","last_change":"2024-09-16T04:42:40.576344+0000","last_active":"2024-09-16T04:42:40.575107+0000","last_peered":"2024-09-16T04:42:40.575107+0000","last_clean":"2024-09-16T04:42:40.575107+0000","last_became_active":"2024-09-16T04:42:18.699582+0000","last_became_peered":"2024-09-16T04:42:18.699582+0000","last_unstale":"2024-09-16T04:42:40.575107+0000","last_undegraded":"2024-09-16T04:42:40.575107+0000","last_fullsized":"2024-09-16T04:42:40.575107+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'15","last_scrub_stamp":"2024-09-16T04:42:26.595325+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.595325+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:30.656305+0000","scrub_duration":967,"objects_trimmed":0,"snaptrim_duration":0.001637009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"8"},{"start":"12","length":"6"}]},{"pgid":"2.3","version":"26'1","reported_seq":142,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.707106+0000","last_change":"2024-09-16T04:41:30.209703+0000","last_active":"2024-09-16T04:42:39.707106+0000","last_peered":"2024-09-16T04:42:39.707106+0000","last_clean":"2024-09-16T04:42:39.707106+0000","last_became_active":"2024-09-16T04:41:27.604318+0000","last_became_peered":"2024-09-16T04:41:27.604318+0000","last_unstale":"2024-09-16T04:42:39.707106+0000","last_undegraded":"2024-09-16T04:42:39.707106+0000","last_fullsized":"2024-09-16T04:42:39.707106+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:42:49.941157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":345,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.474352+0000","last_change":"2024-09-16T04:42:36.062121+0000","last_active":"2024-09-16T04:42:40.474352+0000","last_peered":"2024-09-16T04:42:40.474352+0000","last_clean":"2024-09-16T04:42:40.474352+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:42:40.474352+0000","last_undegraded":"2024-09-16T04:42:40.474352+0000","last_fullsized":"2024-09-16T04:42:40.474352+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:42:36.061866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:42:36.061866+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:37.369149+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"82'27","reported_seq":339,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.514637+0000","last_change":"2024-09-16T04:42:38.673194+0000","last_active":"2024-09-16T04:42:39.514637+0000","last_peered":"2024-09-16T04:42:39.514637+0000","last_clean":"2024-09-16T04:42:39.514637+0000","last_became_active":"2024-09-16T04:42:06.535331+0000","last_became_peered":"2024-09-16T04:42:06.535331+0000","last_unstale":"2024-09-16T04:42:39.514637+0000","last_undegraded":"2024-09-16T04:42:39.514637+0000","last_fullsized":"2024-09-16T04:42:39.514637+0000","mapping_epoch":32,"log_start":"44'8","ondisk_log_start":"44'8","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-09-16T04:42:13.576020+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:13.576020+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":12,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:34.246027+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0.0019395949999999999,"stat_sum":{"num_bytes":2752512,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.9","version":"50'30","reported_seq":335,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.582137+0000","last_change":"2024-09-16T04:42:38.583792+0000","last_active":"2024-09-16T04:42:38.582137+0000","last_peered":"2024-09-16T04:42:38.582137+0000","last_clean":"2024-09-16T04:42:38.582137+0000","last_became_active":"2024-09-16T04:42:06.419535+0000","last_became_peered":"2024-09-16T04:42:06.419535+0000","last_unstale":"2024-09-16T04:42:38.582137+0000","last_undegraded":"2024-09-16T04:42:38.582137+0000","last_fullsized":"2024-09-16T04:42:38.582137+0000","mapping_epoch":32,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'30","last_scrub_stamp":"2024-09-16T04:42:10.554025+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.554025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.985451+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.0019264939999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.a","version":"88'63","reported_seq":363,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.672224+0000","last_change":"2024-09-16T04:42:38.974918+0000","last_active":"2024-09-16T04:42:39.672224+0000","last_peered":"2024-09-16T04:42:39.672224+0000","last_clean":"2024-09-16T04:42:39.672224+0000","last_became_active":"2024-09-16T04:42:18.694273+0000","last_became_peered":"2024-09-16T04:42:18.694273+0000","last_unstale":"2024-09-16T04:42:39.672224+0000","last_undegraded":"2024-09-16T04:42:39.672224+0000","last_fullsized":"2024-09-16T04:42:39.672224+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'41","last_scrub_stamp":"2024-09-16T04:42:25.530017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.530017+0000","objects_scrubbed":5,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:37.491140+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.506247793,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":8,"num_read_kb":482,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.b","version":"87'69","reported_seq":357,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.474046+0000","last_change":"2024-09-16T04:42:38.474680+0000","last_active":"2024-09-16T04:42:38.474046+0000","last_peered":"2024-09-16T04:42:38.474046+0000","last_clean":"2024-09-16T04:42:38.474046+0000","last_became_active":"2024-09-16T04:42:18.688040+0000","last_became_peered":"2024-09-16T04:42:18.688040+0000","last_unstale":"2024-09-16T04:42:38.474046+0000","last_undegraded":"2024-09-16T04:42:38.474046+0000","last_fullsized":"2024-09-16T04:42:38.474046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'58","last_scrub_stamp":"2024-09-16T04:42:25.280206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.280206+0000","objects_scrubbed":6,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:49.638571+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0.001704971,"stat_sum":{"num_bytes":7864320,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":9112,"num_write":4,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.c","version":"92'87","reported_seq":384,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:42:42.922879+0000","last_change":"2024-09-16T04:42:42.926178+0000","last_active":"2024-09-16T04:42:42.922879+0000","last_peered":"2024-09-16T04:42:42.922879+0000","last_clean":"2024-09-16T04:42:42.922879+0000","last_became_active":"2024-09-16T04:42:18.798872+0000","last_became_peered":"2024-09-16T04:42:18.798872+0000","last_unstale":"2024-09-16T04:42:42.922879+0000","last_undegraded":"2024-09-16T04:42:42.922879+0000","last_fullsized":"2024-09-16T04:42:42.922879+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:35.566994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.566994+0000","objects_scrubbed":6,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:55.866860+0000","scrub_duration":743,"objects_trimmed":2,"snaptrim_duration":0.40961999100000002,"stat_sum":{"num_bytes":3457024,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"f"}]},{"pgid":"3.d","version":"88'176","reported_seq":453,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.102868+0000","last_change":"2024-09-16T04:42:39.104900+0000","last_active":"2024-09-16T04:42:39.102868+0000","last_peered":"2024-09-16T04:42:39.102868+0000","last_clean":"2024-09-16T04:42:39.102868+0000","last_became_active":"2024-09-16T04:42:19.196254+0000","last_became_peered":"2024-09-16T04:42:19.196254+0000","last_unstale":"2024-09-16T04:42:39.102868+0000","last_undegraded":"2024-09-16T04:42:39.102868+0000","last_fullsized":"2024-09-16T04:42:39.102868+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'134","last_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_deep_scrub":"74'134","last_deep_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.664276+0000","objects_scrubbed":9,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:34.008505+0000","scrub_duration":1071,"objects_trimmed":0,"snaptrim_duration":0.52758384700000005,"stat_sum":{"num_bytes":8519680,"num_objects":11,"num_object_clones":7,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":3,"num_read":7,"num_read_kb":4921,"num_write":27,"num_write_kb":1936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.e","version":"87'80","reported_seq":367,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.471276+0000","last_change":"2024-09-16T04:42:38.780088+0000","last_active":"2024-09-16T04:42:39.471276+0000","last_peered":"2024-09-16T04:42:39.471276+0000","last_clean":"2024-09-16T04:42:39.471276+0000","last_became_active":"2024-09-16T04:42:18.700111+0000","last_became_peered":"2024-09-16T04:42:18.700111+0000","last_unstale":"2024-09-16T04:42:39.471276+0000","last_undegraded":"2024-09-16T04:42:39.471276+0000","last_fullsized":"2024-09-16T04:42:39.471276+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:34.646929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.646929+0000","objects_scrubbed":6,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:57.121133+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.0021472129999999998,"stat_sum":{"num_bytes":7454720,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.f","version":"90'88","reported_seq":386,"reported_epoch":92,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:42.825291+0000","last_change":"2024-09-16T04:42:42.825449+0000","last_active":"2024-09-16T04:42:42.825291+0000","last_peered":"2024-09-16T04:42:42.825291+0000","last_clean":"2024-09-16T04:42:42.825291+0000","last_became_active":"2024-09-16T04:42:18.793319+0000","last_became_peered":"2024-09-16T04:42:18.793319+0000","last_unstale":"2024-09-16T04:42:42.825291+0000","last_undegraded":"2024-09-16T04:42:42.825291+0000","last_fullsized":"2024-09-16T04:42:42.825291+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'74","last_scrub_stamp":"2024-09-16T04:42:34.345034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.345034+0000","objects_scrubbed":7,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:55.949634+0000","scrub_duration":719,"objects_trimmed":0,"snaptrim_duration":0.210027677,"stat_sum":{"num_bytes":6258688,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":10815,"num_write":5,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"8"},{"start":"12","length":"6"}]},{"pgid":"3.10","version":"88'25","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T04:42:39.570810+0000","last_change":"2024-09-16T04:42:38.981802+0000","last_active":"2024-09-16T04:42:39.570810+0000","last_peered":"2024-09-16T04:42:39.570810+0000","last_clean":"2024-09-16T04:42:39.570810+0000","last_became_active":"2024-09-16T04:42:16.560449+0000","last_became_peered":"2024-09-16T04:42:16.560449+0000","last_unstale":"2024-09-16T04:42:39.570810+0000","last_undegraded":"2024-09-16T04:42:39.570810+0000","last_fullsized":"2024-09-16T04:42:39.570810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:42:24.882865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.882865+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:39.676913+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0.30427622500000001,"stat_sum":{"num_bytes":3162112,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1,14,5,3,15],"acting":[2,7,1,14,5,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.11","version":"82'38","reported_seq":249,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T04:42:38.885148+0000","last_change":"2024-09-16T04:42:38.885676+0000","last_active":"2024-09-16T04:42:38.885148+0000","last_peered":"2024-09-16T04:42:38.885148+0000","last_clean":"2024-09-16T04:42:38.885148+0000","last_became_active":"2024-09-16T04:42:04.623974+0000","last_became_peered":"2024-09-16T04:42:04.623974+0000","last_unstale":"2024-09-16T04:42:38.885148+0000","last_undegraded":"2024-09-16T04:42:38.885148+0000","last_fullsized":"2024-09-16T04:42:38.885148+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'28","last_scrub_stamp":"2024-09-16T04:42:10.158979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.158979+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:34.840621+0000","scrub_duration":855,"objects_trimmed":0,"snaptrim_duration":0.00075688199999999995,"stat_sum":{"num_bytes":3653632,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":5,"num_read_kb":5233,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"3"},{"start":"12","length":"6"}]},{"pgid":"3.12","version":"90'28","reported_seq":234,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T04:42:40.891170+0000","last_change":"2024-09-16T04:42:40.893357+0000","last_active":"2024-09-16T04:42:40.891170+0000","last_peered":"2024-09-16T04:42:40.891170+0000","last_clean":"2024-09-16T04:42:40.891170+0000","last_became_active":"2024-09-16T04:42:04.406048+0000","last_became_peered":"2024-09-16T04:42:04.406048+0000","last_unstale":"2024-09-16T04:42:40.891170+0000","last_undegraded":"2024-09-16T04:42:40.891170+0000","last_fullsized":"2024-09-16T04:42:40.891170+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'12","last_scrub_stamp":"2024-09-16T04:42:10.115913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.115913+0000","objects_scrubbed":1,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:36.115543+0000","scrub_duration":611,"objects_trimmed":1,"snaptrim_duration":0.30616532600000002,"stat_sum":{"num_bytes":1277952,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3745,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,15],"acting":[9,7,8,6,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"8"},{"start":"12","length":"6"}]},{"pgid":"3.13","version":"36'3","reported_seq":207,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T04:42:42.516635+0000","last_change":"2024-09-16T04:42:42.517309+0000","last_active":"2024-09-16T04:42:42.516635+0000","last_peered":"2024-09-16T04:42:42.516635+0000","last_clean":"2024-09-16T04:42:42.516635+0000","last_became_active":"2024-09-16T04:42:04.415210+0000","last_became_peered":"2024-09-16T04:42:04.415210+0000","last_unstale":"2024-09-16T04:42:42.516635+0000","last_undegraded":"2024-09-16T04:42:42.516635+0000","last_fullsized":"2024-09-16T04:42:42.516635+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:42:10.690368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.690368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:24.634246+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.00086424400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,0,4,10],"acting":[11,7,9,13,0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"f"}]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":89341952,"num_objects":96,"num_object_clones":50,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":49,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":7,"num_read":111,"num_read_kb":74745,"num_write":164,"num_write_kb":26496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4505600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":165863424,"data_stored":224899072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1492,"ondisk_log_size":1492,"up":252,"acting":252,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247828,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42908,"kb_used_data":11712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328932,"statfs":{"total":96636764160,"available":96592826368,"internally_reserved":0,"allocated":11993088,"data_stored":16768953,"data_compressed":234400,"data_compressed_allocated":319488,"data_compressed_original":1069056,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247828,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42652,"kb_used_data":11464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329188,"statfs":{"total":96636764160,"available":96593088512,"internally_reserved":0,"allocated":11739136,"data_stored":14931150,"data_compressed":199426,"data_compressed_allocated":274432,"data_compressed_original":548864,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,6,7,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247828,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42772,"kb_used_data":11580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329068,"statfs":{"total":96636764160,"available":96592965632,"internally_reserved":0,"allocated":11857920,"data_stored":15348942,"data_compressed":199426,"data_compressed_allocated":274432,"data_compressed_original":548864,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313235,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":12588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":12890112,"data_stored":16674585,"data_compressed":208503,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313235,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42824,"kb_used_data":11632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329016,"statfs":{"total":96636764160,"available":96592912384,"internally_reserved":0,"allocated":11911168,"data_stored":14237465,"data_compressed":208503,"data_compressed_allocated":286720,"data_compressed_original":573440,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280531,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40744,"kb_used_data":9560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331096,"statfs":{"total":96636764160,"available":96595042304,"internally_reserved":0,"allocated":9789440,"data_stored":12250649,"data_compressed":228356,"data_compressed_allocated":311296,"data_compressed_original":1052672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280530,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44116,"kb_used_data":12920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327724,"statfs":{"total":96636764160,"available":96591589376,"internally_reserved":0,"allocated":13230080,"data_stored":17169401,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,6,7,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42796,"kb_used_data":11608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329044,"statfs":{"total":96636764160,"available":96592941056,"internally_reserved":0,"allocated":11886592,"data_stored":16778465,"data_compressed":199426,"data_compressed_allocated":274432,"data_compressed_original":548864,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280531,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43468,"kb_used_data":12280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328372,"statfs":{"total":96636764160,"available":96592252928,"internally_reserved":0,"allocated":12574720,"data_stored":17886201,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280531,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41964,"kb_used_data":10780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329876,"statfs":{"total":96636764160,"available":96593793024,"internally_reserved":0,"allocated":11038720,"data_stored":13618169,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280530,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41428,"kb_used_data":10240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330412,"statfs":{"total":96636764160,"available":96594341888,"internally_reserved":0,"allocated":10485760,"data_stored":13499385,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42356,"kb_used_data":11168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329484,"statfs":{"total":96636764160,"available":96593391616,"internally_reserved":0,"allocated":11436032,"data_stored":15285241,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280530,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45240,"kb_used_data":14056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326600,"statfs":{"total":96636764160,"available":96590438400,"internally_reserved":0,"allocated":14393344,"data_stored":19926009,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280531,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41128,"kb_used_data":9940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330712,"statfs":{"total":96636764160,"available":96594649088,"internally_reserved":0,"allocated":10178560,"data_stored":14420985,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,5,6,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280530,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33572,"kb_used_data":6544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338268,"statfs":{"total":96636764160,"available":96602386432,"internally_reserved":0,"allocated":6701056,"data_stored":7416844,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280531,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43608,"kb_used_data":12420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328232,"statfs":{"total":96636764160,"available":96592109568,"internally_reserved":0,"allocated":12718080,"data_stored":16661497,"data_compressed":196411,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11829248,"data_stored":15892480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10833920,"data_stored":15204352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8716288,"data_stored":10981376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10858496,"data_stored":16027648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11563008,"data_stored":17145856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9551872,"data_stored":12402688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9474048,"data_stored":12759040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10424320,"data_stored":14544896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13381632,"data_stored":19185664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10539008,"data_stored":13475840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":13680640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":9969664,"data_stored":11386880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":5685248,"data_stored":6676480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":13660160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":11231232,"data_stored":15446016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":12218368,"data_stored":16429056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:42:44.385 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-09-16T04:42:44.385 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-16T04:42:44.396 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:44.397 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:44.412 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:42:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:342: done (4 left) 2024-09-16T04:42:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:343: rollback oid 22 current snap is 27 2024-09-16T04:42:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 22 2024-09-16T04:42:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:344: rollback oid 31 current snap is 27 2024-09-16T04:42:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 23 2024-09-16T04:42:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 37 current snap is 27 2024-09-16T04:42:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:345: seq_num 130 ranges {606208=638976,1245184=671744,1916928=802816,2719744=81920} 2024-09-16T04:42:44.451 INFO:tasks.rados.rados.0.smithi165.stdout:345: writing smithi16530038-37 from 606208 to 1245184 tid 1 2024-09-16T04:42:44.459 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:44.455+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:44.462 INFO:tasks.rados.rados.0.smithi165.stdout:345: writing smithi16530038-37 from 1245184 to 1916928 tid 2 2024-09-16T04:42:44.475 INFO:tasks.rados.rados.0.smithi165.stdout:345: writing smithi16530038-37 from 1916928 to 2719744 tid 3 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:345: writing smithi16530038-37 from 2719744 to 2801664 tid 4 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:346: read oid 17 snap -1 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:346: expect (ObjNum 104 snap 18 seq_num 104) 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:347: rollback oid 19 current snap is 27 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 19 to 22 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:343: finishing rollback tid 1 to smithi16530038-22 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:343: done (8 left) 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:348: snap_create 2024-09-16T04:42:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:338: finishing copy_from racing read to smithi16530038-33 2024-09-16T04:42:44.511 INFO:tasks.rados.rados.0.smithi165.stdout:348: done (8 left) 2024-09-16T04:42:44.511 INFO:tasks.rados.rados.0.smithi165.stdout:349: rollback oid 12 current snap is 28 2024-09-16T04:42:44.511 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 24 2024-09-16T04:42:44.512 INFO:tasks.rados.rados.0.smithi165.stdout:350: rmattr oid 6 current snap is 28 2024-09-16T04:42:44.512 INFO:tasks.rados.rados.0.smithi165.stdout:350: done (9 left) 2024-09-16T04:42:44.512 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 18 current snap is 28 2024-09-16T04:42:44.512 INFO:tasks.rados.rados.0.smithi165.stdout:351: seq_num 131 ranges {1671168=655360,2326528=425984,2752512=638976,3391488=491520} 2024-09-16T04:42:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:351: writing smithi16530038-18 from 1671168 to 2326528 tid 1 2024-09-16T04:42:44.544 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:42:44.550 INFO:tasks.rados.rados.0.smithi165.stdout:351: writing smithi16530038-18 from 2326528 to 2752512 tid 2 2024-09-16T04:42:44.550 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:44.550 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:44.557 INFO:tasks.rados.rados.0.smithi165.stdout:351: writing smithi16530038-18 from 2752512 to 3391488 tid 3 2024-09-16T04:42:44.561 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:44.555+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:351: writing smithi16530038-18 from 3391488 to 3883008 tid 4 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:352: copy_from oid 23 from oid 48 current snap is 28 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:340: finishing rollback tid 1 to smithi16530038-42 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 164 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:340: done (10 left) 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:353: rmattr oid 10 current snap is 28 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:353: done (10 left) 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 43 current snap is 28 2024-09-16T04:42:44.563 INFO:tasks.rados.rados.0.smithi165.stdout:354: seq_num 132 ranges {4145152=671744,4816896=573440,5390336=753664,6144000=229376} 2024-09-16T04:42:44.565 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:42:44.601 INFO:tasks.rados.rados.0.smithi165.stdout:354: writing smithi16530038-43 from 4145152 to 4816896 tid 1 2024-09-16T04:42:44.604 INFO:tasks.rados.rados.0.smithi165.stdout:354: writing smithi16530038-43 from 4816896 to 5390336 tid 2 2024-09-16T04:42:44.609 INFO:tasks.rados.rados.0.smithi165.stdout:354: writing smithi16530038-43 from 5390336 to 6144000 tid 3 2024-09-16T04:42:44.611 INFO:tasks.rados.rados.0.smithi165.stdout:354: writing smithi16530038-43 from 6144000 to 6373376 tid 4 2024-09-16T04:42:44.611 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 13 current snap is 28 2024-09-16T04:42:44.611 INFO:tasks.rados.rados.0.smithi165.stdout:355: seq_num 133 ranges {1671168=688128,2359296=720896,3080192=475136,3555328=360448} 2024-09-16T04:42:44.624 INFO:tasks.rados.rados.0.smithi165.stdout:355: writing smithi16530038-13 from 1671168 to 2359296 tid 1 2024-09-16T04:42:44.628 INFO:tasks.rados.rados.0.smithi165.stdout:355: writing smithi16530038-13 from 2359296 to 3080192 tid 2 2024-09-16T04:42:44.631 INFO:tasks.rados.rados.0.smithi165.stdout:355: writing smithi16530038-13 from 3080192 to 3555328 tid 3 2024-09-16T04:42:44.633 INFO:tasks.rados.rados.0.smithi165.stdout:355: writing smithi16530038-13 from 3555328 to 3915776 tid 4 2024-09-16T04:42:44.633 INFO:tasks.rados.rados.0.smithi165.stdout:356: copy_from oid 47 from oid 11 current snap is 28 2024-09-16T04:42:44.634 INFO:tasks.rados.rados.0.smithi165.stdout:338: finishing copy_from to smithi16530038-33 2024-09-16T04:42:44.634 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 69 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T04:42:44.634 INFO:tasks.rados.rados.0.smithi165.stdout:344: finishing rollback tid 1 to smithi16530038-31 2024-09-16T04:42:44.634 INFO:tasks.rados.rados.0.smithi165.stdout:341: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:44.653 INFO:tasks.rados.rados.0.smithi165.stdout:338: done (12 left) 2024-09-16T04:42:44.653 INFO:tasks.rados.rados.0.smithi165.stdout:341: done (11 left) 2024-09-16T04:42:44.653 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 3 current snap is 28 2024-09-16T04:42:44.653 INFO:tasks.rados.rados.0.smithi165.stdout:357: seq_num 134 ranges {1228800=704512,1933312=475136,2408448=573440,2981888=507904} 2024-09-16T04:42:44.662 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:44.659+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:44.664 INFO:tasks.rados.rados.0.smithi165.stdout:357: writing smithi16530038-3 from 1228800 to 1933312 tid 1 2024-09-16T04:42:44.667 INFO:tasks.rados.rados.0.smithi165.stdout:357: writing smithi16530038-3 from 1933312 to 2408448 tid 2 2024-09-16T04:42:44.672 INFO:tasks.rados.rados.0.smithi165.stdout:357: writing smithi16530038-3 from 2408448 to 2981888 tid 3 2024-09-16T04:42:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:357: writing smithi16530038-3 from 2981888 to 3489792 tid 4 2024-09-16T04:42:44.677 INFO:tasks.rados.rados.0.smithi165.stdout:352: finishing copy_from to smithi16530038-23 2024-09-16T04:42:44.677 INFO:tasks.rados.rados.0.smithi165.stdout:352: got expected ENOENT (src dne) 2024-09-16T04:42:44.677 INFO:tasks.rados.rados.0.smithi165.stdout:352: finishing copy_from racing read to smithi16530038-23 2024-09-16T04:42:44.677 INFO:tasks.rados.rados.0.smithi165.stdout:352: done (11 left) 2024-09-16T04:42:44.677 INFO:tasks.rados.rados.0.smithi165.stdout:358: snap_remove snap 26 2024-09-16T04:42:44.739 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:44.739 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:44.755 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:42:44.763 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:44.759+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:44.864 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:44.858+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:44.875 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:44.875 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:44.892 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:42:44.965 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:44.963+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:45.018 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:45.018 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:45.034 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:42:45.067 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:45.062+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:45.157 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:45.157 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:45.168 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:45.163+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:45.171 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:42:45.269 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:45.267+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:45.337 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:45.337 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:45.351 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:42:45.399 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:45.395+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:45.426 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:45.470 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:45.471 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:45.471 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:45.466+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:45.486 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:42:45.527 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pg_num to 26 2024-09-16T04:42:45.527 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:42:45.531 INFO:tasks.rados.rados.0.smithi165.stdout:358: done (11 left) 2024-09-16T04:42:45.531 INFO:tasks.rados.rados.0.smithi165.stdout:359: snap_create 2024-09-16T04:42:45.531 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 1 to smithi16530038-37 2024-09-16T04:42:45.531 INFO:tasks.rados.rados.0.smithi165.stdout:344: finishing rollback tid 2 to smithi16530038-31 2024-09-16T04:42:45.531 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 55 (ObjNum 83 snap 12 seq_num 83) dirty exists 2024-09-16T04:42:45.531 INFO:tasks.rados.rados.0.smithi165.stdout:339: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 2 to smithi16530038-37 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:349: finishing rollback tid 0 to smithi16530038-12 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:349: finishing rollback tid 1 to smithi16530038-12 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:349: finishing rollback tid 2 to smithi16530038-12 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 77 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:347: finishing rollback tid 1 to smithi16530038-19 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:354: finishing write tid 1 to smithi16530038-43 2024-09-16T04:42:45.540 INFO:tasks.rados.rados.0.smithi165.stdout:354: finishing write tid 2 to smithi16530038-43 2024-09-16T04:42:45.541 INFO:tasks.rados.rados.0.smithi165.stdout:355: finishing write tid 1 to smithi16530038-13 2024-09-16T04:42:45.541 INFO:tasks.rados.rados.0.smithi165.stdout:346: expect (ObjNum 104 snap 18 seq_num 104) 2024-09-16T04:42:45.571 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:45.567+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:45.593 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:45.594 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:45.612 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:45.613 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:45.635 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:45.635 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:45.660 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:45.661 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:45.664 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:45.664 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:45.672 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:45.667+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:45.676 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:45.677 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:45.679 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:42:45.698 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:45.699 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:45.706 INFO:tasks.rados.rados.0.smithi165.stdout:356: finishing copy_from racing read to smithi16530038-47 2024-09-16T04:42:45.706 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing write tid 1 to smithi16530038-3 2024-09-16T04:42:45.706 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing write tid 2 to smithi16530038-3 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing write tid 3 to smithi16530038-3 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing write tid 4 to smithi16530038-3 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing write tid 1 to smithi16530038-18 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing write tid 2 to smithi16530038-18 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing write tid 3 to smithi16530038-18 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing write tid 4 to smithi16530038-18 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:347: finishing rollback tid 2 to smithi16530038-19 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 32 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:354: finishing write tid 3 to smithi16530038-43 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:354: finishing write tid 4 to smithi16530038-43 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:354: finishing write tid 5 to smithi16530038-43 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:355: finishing write tid 2 to smithi16530038-13 2024-09-16T04:42:45.707 INFO:tasks.rados.rados.0.smithi165.stdout:355: finishing write tid 3 to smithi16530038-13 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:355: finishing write tid 4 to smithi16530038-13 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 3 to smithi16530038-37 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing write tid 5 to smithi16530038-18 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:355: finishing write tid 5 to smithi16530038-13 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:356: finishing copy_from to smithi16530038-47 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 112 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 4 to smithi16530038-37 2024-09-16T04:42:45.708 INFO:tasks.rados.rados.0.smithi165.stdout:354: finishing write tid 6 to smithi16530038-43 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:354: oid 43 updating version 0 to 29 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:354: oid 43 version 29 is already newer than 27 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:354: oid 43 updating version 29 to 31 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:354: oid 43 version 31 is already newer than 28 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:354: oid 43 version 31 is already newer than 30 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 31 (ObjNum 132 snap 28 seq_num 132) dirty exists 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:354: left oid 43 (ObjNum 132 snap 28 seq_num 132) 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 5 to smithi16530038-37 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing write tid 5 to smithi16530038-3 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 6 to smithi16530038-37 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 37 updating version 0 to 12 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 37 updating version 12 to 14 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 37 updating version 14 to 16 2024-09-16T04:42:45.709 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 37 version 16 is already newer than 13 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 37 version 16 is already newer than 15 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 16 (ObjNum 130 snap 27 seq_num 130) dirty exists 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:345: left oid 37 (ObjNum 130 snap 27 seq_num 130) 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing write tid 6 to smithi16530038-18 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:351: oid 18 updating version 0 to 152 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:351: oid 18 updating version 152 to 153 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:351: oid 18 version 153 is already newer than 149 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:351: oid 18 version 153 is already newer than 150 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:351: oid 18 version 153 is already newer than 151 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 153 (ObjNum 131 snap 28 seq_num 131) dirty exists 2024-09-16T04:42:45.710 INFO:tasks.rados.rados.0.smithi165.stdout:351: left oid 18 (ObjNum 131 snap 28 seq_num 131) 2024-09-16T04:42:45.717 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:45.717 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:45.737 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:45.748 INFO:tasks.rados.rados.0.smithi165.stdout:355: finishing write tid 6 to smithi16530038-13 2024-09-16T04:42:45.749 INFO:tasks.rados.rados.0.smithi165.stdout:355: oid 13 updating version 0 to 83 2024-09-16T04:42:45.749 INFO:tasks.rados.rados.0.smithi165.stdout:355: oid 13 version 83 is already newer than 82 2024-09-16T04:42:45.749 INFO:tasks.rados.rados.0.smithi165.stdout:355: oid 13 version 83 is already newer than 79 2024-09-16T04:42:45.749 INFO:tasks.rados.rados.0.smithi165.stdout:355: oid 13 version 83 is already newer than 81 2024-09-16T04:42:45.749 INFO:tasks.rados.rados.0.smithi165.stdout:355: oid 13 version 83 is already newer than 80 2024-09-16T04:42:45.749 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 83 (ObjNum 133 snap 28 seq_num 133) dirty exists 2024-09-16T04:42:45.749 INFO:tasks.rados.rados.0.smithi165.stdout:355: left oid 13 (ObjNum 133 snap 28 seq_num 133) 2024-09-16T04:42:45.758 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:45.758 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:45.773 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:45.767+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:45.846 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:45.846 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:45.861 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:42:45.904 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:45.899+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing write tid 6 to smithi16530038-3 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:357: oid 3 updating version 0 to 92 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:357: oid 3 updating version 92 to 95 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:357: oid 3 version 95 is already newer than 94 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:357: oid 3 version 95 is already newer than 93 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:357: oid 3 updating version 95 to 96 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 96 (ObjNum 134 snap 28 seq_num 134) dirty exists 2024-09-16T04:42:45.940 INFO:tasks.rados.rados.0.smithi165.stdout:357: left oid 3 (ObjNum 134 snap 28 seq_num 134) 2024-09-16T04:42:45.941 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:45.935+0000 7f2556130640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 1c after trimming was started 2024-09-16T04:42:45.975 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:45.970+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:46.029 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:46.030 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:46.043 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:46.043 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:46.045 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:42:46.079 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:46.079 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:46.091 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:46.087+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:46.113 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:46.113 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:46.148 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:46.148 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:46.172 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:46.173 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:46.176 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:46.171+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:46.179 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:46.187 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:42:46.209 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:46.210 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:46.238 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:46.239 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:46.277 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:46.271+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:46.360 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:46.360 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:46.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:46.360 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:46.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.360 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 81.980756464999999, 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1755266710000001, 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:42:46.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.362 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374332+0000", 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8059999999999999e-06 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374345+0000", 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2529e-05 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374365+0000", 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0695000000000001e-05 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374383+0000", 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7569000000000001e-05 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374514+0000", 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013070099999999999 2024-09-16T04:42:46.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374544+0000", 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0731000000000001e-05 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.343896+0000", 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.969351224 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344169+0000", 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027390100000000001 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344583+0000", 2024-09-16T04:42:46.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041308600000000001 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.346047+0000", 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014642699999999999 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549695+0000", 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203648463 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549791+0000", 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6137999999999993e-05 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549856+0000", 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.368 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.734340286000005, 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51346886800000002, 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 111 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620747+0000", 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620757+0000", 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0316000000000001e-05 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620778+0000", 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620792+0000", 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4362000000000001e-05 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928510+0000", 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.307717717 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928629+0000", 2024-09-16T04:42:46.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011919 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928822+0000", 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019246199999999999 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930635+0000", 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00181292 2024-09-16T04:42:46.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134065+0000", 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20343018399999999 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134149+0000", 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4635000000000005e-05 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134214+0000", 2024-09-16T04:42:46.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4973000000000003e-05 2024-09-16T04:42:46.376 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.376 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.734269922999999, 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51366009000000001, 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.377 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.378 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.378 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.378 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 112 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620817+0000", 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621035+0000", 2024-09-16T04:42:46.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021731100000000001 2024-09-16T04:42:46.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.381 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.381 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621061+0000", 2024-09-16T04:42:46.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6744999999999999e-05 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621082+0000", 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1061999999999999e-05 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929460+0000", 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.308377346 2024-09-16T04:42:46.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929595+0000", 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013512600000000001 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929798+0000", 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020277199999999999 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932465+0000", 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002667862 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134319+0000", 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20185381599999999 2024-09-16T04:42:46.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134407+0000", 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7894999999999998e-05 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134476+0000", 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8829000000000001e-05 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.733969979999998, 2024-09-16T04:42:46.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51357863199999998, 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 113 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:46.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621118+0000", 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.689e-06 2024-09-16T04:42:46.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621129+0000", 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621164+0000", 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5367999999999997e-05 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621182+0000", 2024-09-16T04:42:46.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8352e-05 2024-09-16T04:42:46.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.390 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930689+0000", 2024-09-16T04:42:46.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.309507015 2024-09-16T04:42:46.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930800+0000", 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000110871 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931026+0000", 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000226293 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.933569+0000", 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002542786 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134566+0000", 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200996272 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134640+0000", 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4412000000000005e-05 2024-09-16T04:42:46.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134694+0000", 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4524000000000002e-05 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.733873232999997, 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51372108500000002, 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.394 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 114 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621214+0000", 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:42:46.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621225+0000", 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1348999999999999e-05 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621245+0000", 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9966000000000002e-05 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621259+0000", 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4102e-05 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931642+0000", 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31038274300000002 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931776+0000", 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133552 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931973+0000", 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197262 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.033893+0000", 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101919751 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134774+0000", 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10088169600000001 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134847+0000", 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2309000000000002e-05 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134934+0000", 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6924999999999999e-05 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.733801424000006, 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51387291300000004, 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.401 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.402 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.402 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.402 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 115 2024-09-16T04:42:46.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.402 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621286+0000", 2024-09-16T04:42:46.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.26e-06 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621492+0000", 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020643500000000001 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621517+0000", 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621534+0000", 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6515999999999999e-05 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932515+0000", 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31098146799999998 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932631+0000", 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011636 2024-09-16T04:42:46.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932814+0000", 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000182406 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134236+0000", 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201422618 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135029+0000", 2024-09-16T04:42:46.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079253199999999998 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135102+0000", 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3286000000000003e-05 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.407 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135157+0000", 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5043000000000001e-05 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.733519606000002, 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.717172162, 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.408 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 116 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621568+0000", 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621606+0000", 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8167999999999998e-05 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621628+0000", 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1742e-05 2024-09-16T04:42:46.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621643+0000", 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5696e-05 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235313+0000", 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61366968300000002 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.412 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235449+0000", 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136036 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235657+0000", 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207405 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237049+0000", 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001392552 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338582+0000", 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101532626 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338670+0000", 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7888999999999998e-05 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338738+0000", 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8769000000000005e-05 2024-09-16T04:42:46.414 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.733411615999998, 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.81671015999999996, 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.415 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 117 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:46.416 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621676+0000", 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4440000000000001e-06 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621686+0000", 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0845e-05 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.417 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621706+0000", 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9168999999999999e-05 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621723+0000", 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7538999999999999e-05 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.418 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236241+0000", 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61451815600000004 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236372+0000", 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013062300000000001 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236574+0000", 2024-09-16T04:42:46.419 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020165600000000001 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237570+0000", 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099642999999999997 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438224+0000", 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20065369 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438317+0000", 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3624999999999999e-05 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438384+0000", 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6983000000000006e-05 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.421 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.422 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:46.422 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 72.060337638999997, 2024-09-16T04:42:46.422 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.508075677, 2024-09-16T04:42:46.424 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.424 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.424 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.424 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.424 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.424 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.424 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 138 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294750+0000", 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294760+0000", 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294779+0000", 2024-09-16T04:42:46.426 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8941999999999999e-05 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294794+0000", 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5038999999999999e-05 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405306+0000", 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11051159300000001 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.427 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405426+0000", 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011977 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405611+0000", 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000185217 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709561+0000", 2024-09-16T04:42:46.428 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394976699999998 2024-09-16T04:42:46.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.429 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.429 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802549+0000", 2024-09-16T04:42:46.429 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092988137999999998 2024-09-16T04:42:46.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802707+0000", 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015861799999999999 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802824+0000", 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116517 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.430 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 72.060267515999996, 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60756662299999997, 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.431 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 139 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294820+0000", 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4330000000000001e-06 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294830+0000", 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0023e-05 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.433 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294850+0000", 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0622e-05 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294864+0000", 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4042e-05 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406110+0000", 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11124526 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406222+0000", 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111871 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406398+0000", 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017598000000000001 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709577+0000", 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30317971500000002 2024-09-16T04:42:46.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902101+0000", 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19252423499999999 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902267+0000", 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016537499999999999 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.436 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902385+0000", 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011806700000000001 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 72.060175388000005, 2024-09-16T04:42:46.437 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60781196900000001, 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 140 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.438 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294912+0000", 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.407e-06 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295040+0000", 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012836600000000001 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295062+0000", 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1682e-05 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295083+0000", 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1109000000000001e-05 2024-09-16T04:42:46.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507008+0000", 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21192486699999999 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507149+0000", 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141042 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507369+0000", 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022046500000000001 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710298+0000", 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202928308 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902525+0000", 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.192226958 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902611+0000", 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6274999999999996e-05 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902722+0000", 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011149 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:46.443 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 72.059947807, 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70811914799999998, 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 141 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.444 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295140+0000", 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295152+0000", 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2053e-05 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295172+0000", 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0282e-05 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.446 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295191+0000", 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8759000000000001e-05 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508486+0000", 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.213295545 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508630+0000", 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014419899999999999 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508824+0000", 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019358600000000001 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710318+0000", 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20149381 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.002981+0000", 2024-09-16T04:42:46.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.29266302500000002 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003144+0000", 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016330500000000001 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003257+0000", 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113064 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.449 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 70.395491078999996, 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51415646199999998, 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.450 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 159 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959597+0000", 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9190000000000002e-06 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959609+0000", 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959629+0000", 2024-09-16T04:42:46.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0035e-05 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959645+0000", 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5786000000000001e-05 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167529+0000", 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.207883768 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.453 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167762+0000", 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023295499999999999 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.168134+0000", 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037181099999999999 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.270830+0000", 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102696683 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473585+0000", 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20275470700000001 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473677+0000", 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1772000000000002e-05 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473751+0000", 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4228999999999997e-05 2024-09-16T04:42:46.455 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.456 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.456 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.456 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.456 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:46.456 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:46.456 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 70.395410317, 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51420228400000001, 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 160 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959677+0000", 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959687+0000", 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959711+0000", 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3811999999999998e-05 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959727+0000", 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:42:46.459 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169193+0000", 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209466241 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169340+0000", 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000147279 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169545+0000", 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000205167 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374117+0000", 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20457175999999999 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473763+0000", 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099646306000000004 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473828+0000", 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4046000000000003e-05 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473878+0000", 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0241000000000002e-05 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.462 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 70.395333038000004, 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51424688600000001, 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 161 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959754+0000", 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:42:46.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959920+0000", 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166088 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959944+0000", 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.465 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959963+0000", 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9176000000000001e-05 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371049+0000", 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41108644500000002 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371185+0000", 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013619899999999999 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371395+0000", 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020912 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.377708+0000", 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006313173 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473891+0000", 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.096182904 2024-09-16T04:42:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473949+0000", 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8016999999999997e-05 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474000+0000", 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 70.395017280999994, 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51552797100000003, 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 162 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960070+0000", 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960077+0000", 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6830000000000008e-06 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960091+0000", 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3653e-05 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960101+0000", 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7629999999999995e-06 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372554+0000", 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41245356599999999 2024-09-16T04:42:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372785+0000", 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023023100000000001 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.373156+0000", 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000371601 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474010+0000", 2024-09-16T04:42:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100854291 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475328+0000", 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001317304 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475477+0000", 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014954899999999999 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475596+0000", 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011913699999999999 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 70.394969052999997, 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51788103100000005, 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 163 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960118+0000", 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960124+0000", 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:42:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960134+0000", 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0115999999999999e-05 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960144+0000", 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374237+0000", 2024-09-16T04:42:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41409377600000002 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374462+0000", 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022478600000000001 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374797+0000", 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000334586 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474018+0000", 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099220947000000004 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477813+0000", 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003795171 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477921+0000", 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010822100000000001 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477998+0000", 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6662999999999994e-05 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 70.394929839, 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51822496799999995, 2024-09-16T04:42:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 164 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960157+0000", 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7399999999999991e-07 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960241+0000", 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3677999999999994e-05 2024-09-16T04:42:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960256+0000", 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5206e-05 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960267+0000", 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1372999999999999e-05 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375650+0000", 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41538323100000002 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375885+0000", 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000234758 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.376293+0000", 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040777599999999999 2024-09-16T04:42:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474024+0000", 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097731440000000003 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478113+0000", 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004088225 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478261+0000", 2024-09-16T04:42:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014825 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478381+0000", 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012005700000000001 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 70.394803765999995, 2024-09-16T04:42:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51862660500000002, 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 165 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.488 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960283+0000", 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960289+0000", 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.49e-06 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960301+0000", 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2585999999999999e-05 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960309+0000", 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:42:46.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472635+0000", 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51232596699999999 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472771+0000", 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136243 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:46.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473033+0000", 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026155699999999999 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474539+0000", 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015065 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478656+0000", 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041164319999999997 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478793+0000", 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013757299999999999 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478909+0000", 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011530400000000001 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.493 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:153 3.3s0 3:c31e3ec6:::smithi16530038-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 60.065504634, 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.511692229, 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 153 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.494 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289583+0000", 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.04e-06 2024-09-16T04:42:46.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289590+0000", 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4899999999999997e-06 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289610+0000", 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9814999999999999e-05 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289625+0000", 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5553000000000002e-05 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.296844+0000", 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072185369999999997 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297045+0000", 2024-09-16T04:42:46.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020125199999999999 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599740+0000", 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.302695037 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:46.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599866+0000", 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012644500000000001 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.801273+0000", 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20140706 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.499 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.500 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:46.500 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:46.500 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:46.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:42:46.502 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:46.495+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:46.535 INFO:tasks.rados.rados.0.smithi165.stdout:339: done (11 left) 2024-09-16T04:42:46.535 INFO:tasks.rados.rados.0.smithi165.stdout:344: done (10 left) 2024-09-16T04:42:46.535 INFO:tasks.rados.rados.0.smithi165.stdout:345: done (9 left) 2024-09-16T04:42:46.535 INFO:tasks.rados.rados.0.smithi165.stdout:346: done (8 left) 2024-09-16T04:42:46.535 INFO:tasks.rados.rados.0.smithi165.stdout:347: done (7 left) 2024-09-16T04:42:46.535 INFO:tasks.rados.rados.0.smithi165.stdout:349: done (6 left) 2024-09-16T04:42:46.535 INFO:tasks.rados.rados.0.smithi165.stdout:351: done (5 left) 2024-09-16T04:42:46.536 INFO:tasks.rados.rados.0.smithi165.stdout:354: done (4 left) 2024-09-16T04:42:46.536 INFO:tasks.rados.rados.0.smithi165.stdout:355: done (3 left) 2024-09-16T04:42:46.536 INFO:tasks.rados.rados.0.smithi165.stdout:356: done (2 left) 2024-09-16T04:42:46.536 INFO:tasks.rados.rados.0.smithi165.stdout:357: done (1 left) 2024-09-16T04:42:46.536 INFO:tasks.rados.rados.0.smithi165.stdout:359: done (0 left) 2024-09-16T04:42:46.536 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 9 current snap is 29 2024-09-16T04:42:46.536 INFO:tasks.rados.rados.0.smithi165.stdout:360: seq_num 135 ranges {147456=720896,868352=622592,1490944=688128,2179072=245760} 2024-09-16T04:42:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:360: writing smithi16530038-9 from 147456 to 868352 tid 1 2024-09-16T04:42:46.547 INFO:tasks.rados.rados.0.smithi165.stdout:360: writing smithi16530038-9 from 868352 to 1490944 tid 2 2024-09-16T04:42:46.554 INFO:tasks.rados.rados.0.smithi165.stdout:360: writing smithi16530038-9 from 1490944 to 2179072 tid 3 2024-09-16T04:42:46.556 INFO:tasks.rados.rados.0.smithi165.stdout:360: writing smithi16530038-9 from 2179072 to 2424832 tid 4 2024-09-16T04:42:46.556 INFO:tasks.rados.rados.0.smithi165.stdout:361: snap_remove snap 22 2024-09-16T04:42:46.603 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:46.599+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:46.636 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:46.636 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:46.636 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:46.636 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:46.651 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:42:46.704 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:46.699+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:46.805 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:46.799+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:46.817 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:46.817 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:46.817 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:46.817 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:46.817 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:46.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:42:46.906 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:46.903+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:46.958 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:108 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 61.593158023000001, 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60644887800000002, 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:46.959 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 108 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:46.960 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362769+0000", 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.364e-06 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362776+0000", 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9789999999999996e-06 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.961 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362795+0000", 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8569000000000001e-05 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362811+0000", 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6084999999999999e-05 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369499+0000", 2024-09-16T04:42:46.962 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066885180000000001 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369745+0000", 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024616900000000002 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669288+0000", 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29954281100000002 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.963 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669425+0000", 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013708200000000001 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.969217+0000", 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29979130100000001 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:507 3.es0 3:7777e846:::smithi16530038-17:head [copy-from ver 21] snapc 10=[10,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.684103084999997, 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.526543015, 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:46.965 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.966 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 507 2024-09-16T04:42:46.966 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.966 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:46.966 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.966 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:46.966 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:46.967 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.967 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.967 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.967 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:46.967 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:46.967 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271824+0000", 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7990000000000001e-06 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271832+0000", 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.968 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271852+0000", 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9194999999999998e-05 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.371991+0000", 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10013957800000001 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372175+0000", 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000183325 2024-09-16T04:42:46.969 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372198+0000", 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3306e-05 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472740+0000", 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100542016 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472902+0000", 2024-09-16T04:42:46.970 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016220399999999999 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692422+0000", 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21951984699999999 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.796322+0000", 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10389983699999999 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.971 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798260+0000", 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001938103 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798366+0000", 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010582900000000001 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.972 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:582 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.616557028999999, 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69089629399999997, 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 582 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.973 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339370+0000", 2024-09-16T04:42:46.974 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339374+0000", 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0799999999999999e-06 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339387+0000", 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3464e-05 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.975 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.976 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339414+0000", 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7223e-05 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446804+0000", 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.107389545 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.977 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446957+0000", 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000153389 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727239+0000", 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28028220199999998 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727462+0000", 2024-09-16T04:42:46.978 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022226099999999999 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.030265+0000", 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280332199999999 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:613 3.es0 3:7777e846:::smithi16530038-17:head [append 0~589824] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.595706540999998, 2024-09-16T04:42:46.979 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55445315100000003, 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 613 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:46.980 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360223+0000", 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3730000000000001e-06 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:46.981 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361140+0000", 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00091743300000000005 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361178+0000", 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7691999999999998e-05 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361197+0000", 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.891e-05 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.982 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361351+0000", 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015419300000000001 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361371+0000", 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9726000000000001e-05 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366548+0000", 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051778730000000004 2024-09-16T04:42:46.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366714+0000", 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000165329 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.369378+0000", 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026643159999999999 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:46.984 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.372406+0000", 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003027891 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914520+0000", 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54211425499999999 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914672+0000", 2024-09-16T04:42:46.985 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015216000000000001 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:614 3.es0 3:7777e846:::smithi16530038-17:head [append 0~540672] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.591733495, 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55075342599999999, 2024-09-16T04:42:46.986 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 614 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.987 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364194+0000", 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.26e-06 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365041+0000", 2024-09-16T04:42:46.988 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00084621499999999999 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365073+0000", 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2719000000000003e-05 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465237+0000", 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100163671 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.989 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465392+0000", 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015528300000000001 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465730+0000", 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000338005 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.467540+0000", 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001809583 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.990 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.469612+0000", 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020719760000000001 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914847+0000", 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.44523504899999999 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914946+0000", 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8665000000000005e-05 2024-09-16T04:42:46.991 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:616 3.es0 3:7777e846:::smithi16530038-17:head [setxattr _header (57) in=7b] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566649+0000", 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.389276336999998, 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50685167900000005, 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:46.992 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 616 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566649+0000", 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566649+0000", 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:46.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566652+0000", 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1600000000000001e-06 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566663+0000", 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1326e-05 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566681+0000", 2024-09-16T04:42:46.994 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8496e-05 2024-09-16T04:42:46.995 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.995 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.995 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:46.995 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566693+0000", 2024-09-16T04:42:46.995 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1143e-05 2024-09-16T04:42:46.995 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.995 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.577966+0000", 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011273350999999999 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.578144+0000", 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017772299999999999 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:46.996 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.578890+0000", 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00074591299999999996 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579957+0000", 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010670860000000001 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.073307+0000", 2024-09-16T04:42:46.997 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49335071800000002 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.073501+0000", 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019376300000000001 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:617 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.389219365999999, 2024-09-16T04:42:46.998 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71993709900000002, 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 617 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:46.999 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566707+0000", 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.001 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566713+0000", 2024-09-16T04:42:47.001 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.784e-06 2024-09-16T04:42:47.001 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.001 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.001 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.001 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566725+0000", 2024-09-16T04:42:47.001 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.173e-05 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566732+0000", 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6860000000000006e-06 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579439+0000", 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012706935000000001 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.002 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579759+0000", 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032009600000000002 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074376+0000", 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49461618299999999 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074520+0000", 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014481800000000001 2024-09-16T04:42:47.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.286643+0000", 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212123017 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:693 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~507904] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.413535+0000", 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.542390401999999, 2024-09-16T04:42:47.004 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50096031799999996, 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 693 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.413535+0000", 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.413535+0000", 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.413538+0000", 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6299999999999998e-06 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.414473+0000", 2024-09-16T04:42:47.006 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00093524999999999999 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.414501+0000", 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8028999999999999e-05 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.414517+0000", 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5469e-05 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.007 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.492165+0000", 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077648791999999994 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.492366+0000", 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020003299999999999 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.494023+0000", 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016577479999999999 2024-09-16T04:42:47.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502358+0000", 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0083349870000000003 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.914240+0000", 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41188197999999998 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.914496+0000", 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025539999999999997 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:694 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~655360] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.423435+0000", 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.532490649, 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.491904444, 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.010 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 694 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.011 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.423435+0000", 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.423435+0000", 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.423437+0000", 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2929999999999999e-06 2024-09-16T04:42:47.012 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424595+0000", 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011580270000000001 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424628+0000", 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2498e-05 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424643+0000", 2024-09-16T04:42:47.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5271999999999999e-05 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.497750+0000", 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073106666000000001 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.497889+0000", 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000139612 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.014 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.499106+0000", 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012162690000000001 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502368+0000", 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003261917 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915109+0000", 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41274159799999999 2024-09-16T04:42:47.015 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915340+0000", 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000230292 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:696 3.10s0 3:0fc9379d:::smithi16530038-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.531217131999998, 2024-09-16T04:42:47.016 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69507003199999995, 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 696 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.017 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424713+0000", 2024-09-16T04:42:47.018 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5599999999999998e-06 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424724+0000", 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0906e-05 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424732+0000", 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6499999999999996e-06 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.019 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502418+0000", 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077686693000000001 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502530+0000", 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000111644 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917065+0000", 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41453470199999998 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.020 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917391+0000", 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032674700000000001 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.119779+0000", 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202387337 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.021 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:709 3.21s0 3:8583b9f2:::smithi16530038-39:head [copy-from ver 2] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.660672+0000", 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.295253433999999, 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.504571824, 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 709 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.022 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660672+0000", 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660672+0000", 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.023 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660677+0000", 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7269999999999998e-06 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660693+0000", 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6276000000000001e-05 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660731+0000", 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7840999999999999e-05 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.024 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660755+0000", 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4249000000000001e-05 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.660932+0000", 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000176868 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.661230+0000", 2024-09-16T04:42:47.025 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00029770099999999999 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.820627+0000", 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15939671699999999 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.913914+0000", 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.093287503999999993 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.026 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.165124+0000", 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.25120952899999999 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.165244+0000", 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120412 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:805 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:47.027 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.4508623939999996, 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51150757400000002, 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 805 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.028 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505067+0000", 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8139999999999998e-06 2024-09-16T04:42:47.029 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505082+0000", 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4453e-05 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505124+0000", 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2849999999999998e-05 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.030 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505148+0000", 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3062999999999999e-05 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611642+0000", 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106494218 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611801+0000", 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000159338 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.031 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.613368+0000", 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001566462 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.711952+0000", 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098584630000000006 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016439+0000", 2024-09-16T04:42:47.032 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30448657000000001 2024-09-16T04:42:47.033 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.033 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.033 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.033 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016571+0000", 2024-09-16T04:42:47.033 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013217599999999999 2024-09-16T04:42:47.033 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.036 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:806 3.10s0 3:0fc9379d:::smithi16530038-35:head [rollback 11] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.450729065, 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89166837600000004, 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.037 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 806 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:42:47.038 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505198+0000", 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7239999999999999e-06 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505210+0000", 2024-09-16T04:42:47.039 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1688999999999999e-05 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505232+0000", 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1903999999999999e-05 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505249+0000", 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7036000000000001e-05 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.040 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017365+0000", 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.512115616 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017525+0000", 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016080200000000001 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.018648+0000", 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011226109999999999 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.041 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021794+0000", 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031461660000000002 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396721+0000", 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.37492636899999998 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396865+0000", 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014445899999999999 2024-09-16T04:42:47.042 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:807 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.4506477420000001, 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89196936699999996, 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.043 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 807 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505279+0000", 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505285+0000", 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0399999999999998e-06 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505308+0000", 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2595999999999998e-05 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505323+0000", 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5342999999999999e-05 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019548+0000", 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51422475999999995 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019822+0000", 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027388100000000002 2024-09-16T04:42:47.046 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021062+0000", 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001239614 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.122291+0000", 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101229427 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397057+0000", 2024-09-16T04:42:47.047 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27476627300000001 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397247+0000", 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000190096 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:809 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~540672] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.4386333310000001, 2024-09-16T04:42:47.048 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70063091600000005, 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 809 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517297+0000", 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8899999999999998e-06 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518377+0000", 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010801750000000001 2024-09-16T04:42:47.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518421+0000", 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3114000000000001e-05 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518443+0000", 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2782e-05 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518619+0000", 2024-09-16T04:42:47.051 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017519200000000001 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518647+0000", 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8053999999999999e-05 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698939+0000", 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180292338 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.052 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.699242+0000", 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000302911 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.702023+0000", 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027812560000000002 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813606+0000", 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11158314900000001 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217721+0000", 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40411436000000001 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217923+0000", 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000202695 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:810 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~720896] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:42:47.054 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.4326283530000001, 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81676015000000002, 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 810 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:42:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523300+0000", 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1140000000000002e-06 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.056 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524411+0000", 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001110307 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524443+0000", 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2067000000000003e-05 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524458+0000", 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5223e-05 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.057 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524644+0000", 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018582900000000001 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524656+0000", 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1705999999999999e-05 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.704974+0000", 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180318379 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.058 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.705144+0000", 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017024499999999999 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.706863+0000", 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00171842 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813610+0000", 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10674731 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.339927+0000", 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52631677099999996 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340058+0000", 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130779 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:811 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~786432] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:47.060 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.4259758769999999, 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81051717700000003, 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 811 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.061 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529953+0000", 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.824e-06 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531687+0000", 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017341699999999999 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531721+0000", 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3756e-05 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531744+0000", 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531894+0000", 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000150505 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531914+0000", 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0137e-05 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.064 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808146+0000", 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.276231536 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808322+0000", 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000175899 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.810020+0000", 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016987230000000001 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813806+0000", 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0037855419999999998 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340306+0000", 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52649966299999995 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340467+0000", 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016146800000000001 2024-09-16T04:42:47.066 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:812 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~49152] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.42414328, 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80902035800000005, 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.067 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 812 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531784+0000", 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.796e-06 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531896+0000", 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011207 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531918+0000", 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1412000000000001e-05 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531934+0000", 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6663999999999998e-05 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698290+0000", 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16635518999999999 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698310+0000", 2024-09-16T04:42:47.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0587000000000001e-05 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811428+0000", 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11311787800000001 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811601+0000", 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017340500000000001 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.812624+0000", 2024-09-16T04:42:47.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001022659 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815470+0000", 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002845752 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340654+0000", 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52518383899999999 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340803+0000", 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014910600000000001 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:813 3.22s0 3:456a3a59:::smithi16530038-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.4239647890000002, 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80925618399999999, 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:47.073 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 813 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:42:47.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531962+0000", 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4339999999999999e-06 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531969+0000", 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7329999999999997e-06 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531987+0000", 2024-09-16T04:42:47.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7467000000000001e-05 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632132+0000", 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10014582499999999 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813864+0000", 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18173177200000001 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814024+0000", 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016021500000000001 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814762+0000", 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00073760799999999995 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.816600+0000", 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001837679 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341069+0000", 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52446895500000001 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341217+0000", 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000148496 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:814 3.22s0 3:456a3a59:::smithi16530038-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:47.078 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.3237599470000001, 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.878598294, 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 814 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:42:47.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632168+0000", 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632177+0000", 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9500000000000007e-06 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632196+0000", 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9296e-05 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632207+0000", 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.137e-05 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815302+0000", 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18309505200000001 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815453+0000", 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000151065 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342052+0000", 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52659867599999999 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342192+0000", 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013975200000000001 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.510764+0000", 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.168572213 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.083 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:47.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:42:47.085 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:47.079+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:47.186 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:47.182+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:47.255 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:47.255 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:47.256 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:47.256 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:47.270 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:42:47.288 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:47.283+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:47.389 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:47.383+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:47.392 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:47.392 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:47.392 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:47.392 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:47.392 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:47.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:42:47.489 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:47.483+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:47.526 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:47.526 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:47.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:47.526 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 55.112782398, 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20191694600000001, 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409856+0000", 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409917+0000", 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1023999999999997e-05 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409958+0000", 2024-09-16T04:42:47.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0945000000000002e-05 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510088+0000", 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100130017 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510247+0000", 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015866599999999999 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510287+0000", 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0330999999999997e-05 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611689+0000", 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101401328 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611771+0000", 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2582000000000003e-05 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.531 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 37.051232564999999, 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246694400000001, 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.532 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471405+0000", 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471443+0000", 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7605000000000003e-05 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471463+0000", 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0452999999999999e-05 2024-09-16T04:42:47.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.571565+0000", 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100102105 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671828+0000", 2024-09-16T04:42:47.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026227 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671853+0000", 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5826e-05 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673802+0000", 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019481730000000001 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673871+0000", 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9624999999999998e-05 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 69/67 ECSubWrite(tid=147, reqid=client.4728.0:574, at_version=69'46, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.406141+0000", 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.116496220999998, 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.119323211, 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.537 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406141+0000", 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406141+0000", 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406142+0000", 2024-09-16T04:42:47.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:42:47.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406184+0000", 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1356000000000002e-05 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406213+0000", 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9592999999999999e-05 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.406230+0000", 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6129000000000001e-05 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.522241+0000", 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.116011266 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.522268+0000", 2024-09-16T04:42:47.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6837000000000001e-05 2024-09-16T04:42:47.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.542 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.542 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.525400+0000", 2024-09-16T04:42:47.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031320190000000002 2024-09-16T04:42:47.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.525464+0000", 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4115000000000005e-05 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 69/67 ECSubWrite(tid=149, reqid=client.4728.0:576, at_version=69'48, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.011691703, 2024-09-16T04:42:47.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30202165600000003, 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:47.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510947+0000", 2024-09-16T04:42:47.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.95e-06 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510983+0000", 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6198999999999997e-05 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.511014+0000", 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1112000000000003e-05 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.611120+0000", 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100105729 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811521+0000", 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040100599999999 2024-09-16T04:42:47.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811553+0000", 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1958999999999997e-05 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812884+0000", 2024-09-16T04:42:47.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013312700000000001 2024-09-16T04:42:47.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812967+0000", 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2430999999999997e-05 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 70/67 ECSubWrite(tid=155, reqid=osd.4.0:153, at_version=0'0, trim_to=0'0, roll_forward_to=70'52))", 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.956771202999999, 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137054300000001, 2024-09-16T04:42:47.550 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565867+0000", 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.84e-06 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565893+0000", 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5715999999999999e-05 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565933+0000", 2024-09-16T04:42:47.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0358999999999998e-05 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666040+0000", 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010693499999999 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666191+0000", 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015013299999999999 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666251+0000", 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0615000000000002e-05 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767150+0000", 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100898584 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767236+0000", 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6360999999999995e-05 2024-09-16T04:42:47.554 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=59, reqid=osd.8.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 23.019681383999998, 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26917078799999999, 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502956+0000", 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3e-07 2024-09-16T04:42:47.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502980+0000", 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3802e-05 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.503017+0000", 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7082999999999997e-05 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603116+0000", 2024-09-16T04:42:47.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098623 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603238+0000", 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012259700000000001 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603255+0000", 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6762999999999999e-05 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772084+0000", 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16882903799999999 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772126+0000", 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2151999999999999e-05 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=97, reqid=client.4728.0:613, at_version=74'68, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:47.559 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.955692565, 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20513274000000001, 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566945+0000", 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.03e-07 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567199+0000", 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025406099999999998 2024-09-16T04:42:47.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567228+0000", 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9589000000000001e-05 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567244+0000", 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6027000000000001e-05 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567424+0000", 2024-09-16T04:42:47.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000180006 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567456+0000", 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1251999999999998e-05 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772018+0000", 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20456203000000001 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772077+0000", 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9372e-05 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=98, reqid=client.4728.0:614, at_version=74'69, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.955352046000002, 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.205026973, 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.564 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567286+0000", 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567509+0000", 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022361500000000001 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567536+0000", 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7063000000000002e-05 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567555+0000", 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669312+0000", 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101756135 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669353+0000", 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1069000000000003e-05 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772248+0000", 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102895556 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772312+0000", 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3559000000000006e-05 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.568 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=60, reqid=client.4728.0:618, at_version=74'51, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.605149+0000", 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.917487655999999, 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.167020959, 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605149+0000", 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605149+0000", 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605151+0000", 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4729999999999999e-06 2024-09-16T04:42:47.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605196+0000", 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5735999999999997e-05 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605235+0000", 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8209e-05 2024-09-16T04:42:47.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605259+0000", 2024-09-16T04:42:47.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4085999999999999e-05 2024-09-16T04:42:47.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605435+0000", 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017617300000000001 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.605455+0000", 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9882e-05 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772138+0000", 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16668382400000001 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772170+0000", 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1575999999999997e-05 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.574 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=61, reqid=client.4728.0:619, at_version=74'52, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.607451+0000", 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.915185738000002, 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.164789196, 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.575 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607451+0000", 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607451+0000", 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607453+0000", 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.306e-06 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607501+0000", 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7691999999999997e-05 2024-09-16T04:42:47.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607549+0000", 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8222e-05 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607572+0000", 2024-09-16T04:42:47.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3240000000000001e-05 2024-09-16T04:42:47.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607736+0000", 2024-09-16T04:42:47.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016364299999999999 2024-09-16T04:42:47.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.607769+0000", 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3036e-05 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772175+0000", 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.164405632 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772240+0000", 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5425000000000005e-05 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 79/37 ECSubWrite(tid=59, reqid=osd.15.0:59, at_version=79'48, trim_to=0'0, roll_forward_to=78'46))", 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 18.043944002, 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.119802273, 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478693+0000", 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2300000000000002e-07 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478737+0000", 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4024999999999999e-05 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478764+0000", 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6838999999999998e-05 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.478783+0000", 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8702000000000002e-05 2024-09-16T04:42:47.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.596719+0000", 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.117936155 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.596737+0000", 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7915999999999999e-05 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.598372+0000", 2024-09-16T04:42:47.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001635372 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.598495+0000", 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000122841 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.5s6 79/37 ECSubWrite(tid=60, reqid=osd.15.0:60, at_version=79'50, trim_to=0'0, roll_forward_to=78'46))", 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.481265+0000", 2024-09-16T04:42:47.587 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 18.041371706, 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.118018944, 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481265+0000", 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481265+0000", 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481267+0000", 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6729999999999999e-06 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481303+0000", 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6358e-05 2024-09-16T04:42:47.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481333+0000", 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9869000000000002e-05 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.481349+0000", 2024-09-16T04:42:47.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6039e-05 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.597457+0000", 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.116108302 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.597481+0000", 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4148999999999999e-05 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.599238+0000", 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017566159999999999 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.599284+0000", 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5938e-05 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=74, reqid=client.4728.0:707, at_version=0'0, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:47.592 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.15036293, 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202706724, 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:47.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372276+0000", 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.782e-06 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372297+0000", 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0907000000000002e-05 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372320+0000", 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3759000000000001e-05 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372333+0000", 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2719e-05 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472559+0000", 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10022637399999999 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472570+0000", 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0351e-05 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574932+0000", 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10236213700000001 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574981+0000", 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8695e-05 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=75, reqid=client.4728.0:708, at_version=79'59, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:42:47.597 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.147944529, 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20032509600000001, 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374694+0000", 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374738+0000", 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3016000000000003e-05 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374782+0000", 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4709999999999997e-05 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374813+0000", 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0700000000000001e-05 2024-09-16T04:42:47.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473059+0000", 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098245669999999993 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473071+0000", 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.24e-05 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574989+0000", 2024-09-16T04:42:47.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10191757 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575017+0000", 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8773e-05 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 80/67 ECSubWrite(tid=114, reqid=osd.2.0:114, at_version=80'75, trim_to=0'0, roll_forward_to=79'73))", 2024-09-16T04:42:47.602 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.128793644000002, 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18121299299999999, 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393844+0000", 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5000000000000003e-07 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393900+0000", 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6436e-05 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393925+0000", 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4703999999999999e-05 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393948+0000", 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3173e-05 2024-09-16T04:42:47.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473572+0000", 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079623997000000002 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473583+0000", 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1327e-05 2024-09-16T04:42:47.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575021+0000", 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101437639 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575056+0000", 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5166999999999997e-05 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.608 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=179, reqid=client.4728.0:716, at_version=81'60, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:42:47.608 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.982333411000001, 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20327186, 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540304+0000", 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.73e-07 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540825+0000", 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00052150999999999999 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540843+0000", 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8321e-05 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540858+0000", 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.452e-05 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.741263+0000", 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040504000000001 2024-09-16T04:42:47.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.741281+0000", 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7946999999999999e-05 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.743496+0000", 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0022152199999999999 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.743575+0000", 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9128999999999994e-05 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=180, reqid=client.4728.0:717, at_version=81'61, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.975952917000001, 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29802481800000002, 2024-09-16T04:42:47.613 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546685+0000", 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4100000000000001e-06 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547131+0000", 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00044599499999999997 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547146+0000", 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5145e-05 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547155+0000", 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8710000000000003e-06 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.742419+0000", 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.195263674 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.742437+0000", 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.84e-05 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.844629+0000", 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102192143 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.844709+0000", 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9179999999999997e-05 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=181, reqid=client.4728.0:718, at_version=81'62, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.650122+0000", 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.872515095000001, 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19530381299999999, 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650122+0000", 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650122+0000", 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650124+0000", 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.514e-06 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650371+0000", 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024698100000000001 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650415+0000", 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3943000000000003e-05 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650442+0000", 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.656e-05 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.843720+0000", 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19327802499999999 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.843736+0000", 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6691e-05 2024-09-16T04:42:47.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.845381+0000", 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016446239999999999 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.845425+0000", 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4474999999999997e-05 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 90/69 ECSubWrite(tid=84, reqid=osd.8.0:84, at_version=90'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.9303371870000001, 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19829587900000001, 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592301+0000", 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5430000000000001e-06 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592337+0000", 2024-09-16T04:42:47.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6075999999999998e-05 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592365+0000", 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7863000000000001e-05 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592380+0000", 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.504e-05 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.788683+0000", 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19630244899999999 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.788698+0000", 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5113000000000001e-05 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.790504+0000", 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018063 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.790595+0000", 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1495000000000004e-05 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.629 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s6 92/55 ECSubWrite(tid=97, reqid=osd.5.0:97, at_version=92'31, trim_to=0'0, roll_forward_to=90'29))", 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.9187835719999997, 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.203223405, 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603854+0000", 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6700000000000003e-07 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603919+0000", 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5079000000000005e-05 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603953+0000", 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4215999999999999e-05 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603970+0000", 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6626999999999999e-05 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.804328+0000", 2024-09-16T04:42:47.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20035859 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.804344+0000", 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5252999999999999e-05 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.807006+0000", 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026628329999999999 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:47.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.807077+0000", 2024-09-16T04:42:47.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0140000000000003e-05 2024-09-16T04:42:47.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.634 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.634 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.634 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:47.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:42:47.636 INFO:tasks.rados.rados.0.smithi165.stdout:361: done (1 left) 2024-09-16T04:42:47.636 INFO:tasks.rados.rados.0.smithi165.stdout:362: snap_create 2024-09-16T04:42:47.637 INFO:tasks.rados.rados.0.smithi165.stdout:360: finishing write tid 1 to smithi16530038-9 2024-09-16T04:42:47.637 INFO:tasks.rados.rados.0.smithi165.stdout:360: finishing write tid 2 to smithi16530038-9 2024-09-16T04:42:47.637 INFO:tasks.rados.rados.0.smithi165.stdout:360: finishing write tid 3 to smithi16530038-9 2024-09-16T04:42:47.637 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:42:47.638 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:47.631+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:47.666 INFO:tasks.rados.rados.0.smithi165.stdout:360: finishing write tid 4 to smithi16530038-9 2024-09-16T04:42:47.736 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:47.730+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:47.819 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:47.819 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:47.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.819 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 97/67 ECSubWrite(tid=255, reqid=osd.11.0:253, at_version=0'0, trim_to=0'0, roll_forward_to=97'94))", 2024-09-16T04:42:47.819 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:47.766332+0000", 2024-09-16T04:42:47.819 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.052152626000000001, 2024-09-16T04:42:47.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.052194722999999998, 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:47.766332+0000", 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:47.766332+0000", 2024-09-16T04:42:47.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:47.766333+0000", 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.497e-06 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:47.766355+0000", 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2197000000000001e-05 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:47.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:47.766402+0000", 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6958e-05 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:47.766415+0000", 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3053e-05 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 1 2024-09-16T04:42:47.822 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:47.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:42:47.878 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:47.875+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:47.938 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:47.935+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:47.958 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:47.958 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:47.958 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:47.959 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:47.959 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:47.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:42:48.038 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:48.034+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:48.139 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:48.135+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:48.148 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:48.148 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:48.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:48.148 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:48.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.148 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=17, reqid=client.4728.0:139, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:48.148 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 62.058120692999999, 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.203994169, 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086851+0000", 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.086853+0000", 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6000000000000001e-06 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087506+0000", 2024-09-16T04:42:48.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00065279900000000002 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087552+0000", 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5840999999999997e-05 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.087581+0000", 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9354999999999999e-05 2024-09-16T04:42:48.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187852+0000", 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100270242 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.187866+0000", 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4286e-05 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290791+0000", 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102925166 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290845+0000", 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3879999999999999e-05 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.9s6 36/32 ECSubWrite(tid=18, reqid=client.4728.0:140, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 62.055435287999998, 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27450581499999999, 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089536+0000", 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089538+0000", 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0370000000000001e-06 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089605+0000", 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6268999999999999e-05 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.089663+0000", 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8257000000000001e-05 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.189764+0000", 2024-09-16T04:42:48.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100101525 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290404+0000", 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10063934500000001 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.290428+0000", 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4782000000000001e-05 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.363982+0000", 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073553303 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.364042+0000", 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0297000000000002e-05 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/32 ECSubWrite(tid=62, reqid=osd.7.0:61, at_version=0'0, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T04:42:48.160 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 53.637913486999999, 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20295831, 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507058+0000", 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507061+0000", 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5059999999999998e-06 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507088+0000", 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7764999999999999e-05 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.507122+0000", 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3671999999999999e-05 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.607262+0000", 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100139588 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707564+0000", 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100302577 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.707595+0000", 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0482000000000001e-05 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.709942+0000", 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002347671 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:54.710016+0000", 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4048999999999995e-05 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/37 ECSubWrite(tid=84, reqid=client.4728.0:380, at_version=49'15, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 47.940209271000001, 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20242805, 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.166 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204762+0000", 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.204766+0000", 2024-09-16T04:42:48.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.523e-06 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205263+0000", 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00049713700000000003 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205312+0000", 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8908e-05 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.205338+0000", 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6602e-05 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207188+0000", 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018498200000000001 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207215+0000", 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6322e-05 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407111+0000", 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.199896194 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.407190+0000", 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9543999999999996e-05 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/37 ECSubWrite(tid=85, reqid=client.4728.0:381, at_version=49'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T04:42:48.171 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 47.938341794999999, 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20298417399999999, 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206630+0000", 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206631+0000", 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206948+0000", 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031706300000000002 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.206984+0000", 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6256e-05 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.207008+0000", 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3726e-05 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.405710+0000", 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19870283399999999 2024-09-16T04:42:48.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.405729+0000", 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8116000000000001e-05 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.409550+0000", 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0038217580000000002 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:00.409614+0000", 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3621000000000006e-05 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 58/37 ECSubWrite(tid=116, reqid=client.4728.0:454, at_version=58'26, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.740983301, 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23856791699999999, 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.176 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403988+0000", 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.403990+0000", 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9769999999999999e-06 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404088+0000", 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7986999999999995e-05 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404119+0000", 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0975000000000001e-05 2024-09-16T04:42:48.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404135+0000", 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5874000000000001e-05 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.540429+0000", 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13629391299999999 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.540454+0000", 2024-09-16T04:42:48.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5075999999999998e-05 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642480+0000", 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102026085 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642556+0000", 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6030000000000002e-05 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 58/37 ECSubWrite(tid=117, reqid=client.4728.0:455, at_version=58'27, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.740808565999998, 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23846093700000001, 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:48.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404163+0000", 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404164+0000", 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9299999999999998e-07 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404183+0000", 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9066e-05 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404196+0000", 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2999999999999999e-05 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.404205+0000", 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4229999999999994e-06 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.541363+0000", 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.137157952 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.541386+0000", 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3028999999999999e-05 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642560+0000", 2024-09-16T04:42:48.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101174403 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.642624+0000", 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3471000000000003e-05 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 58/32 ECSubWrite(tid=116, reqid=client.4728.0:462, at_version=58'44, trim_to=0'0, roll_forward_to=57'42))", 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.308363645, 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20377977999999999, 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836608+0000", 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836610+0000", 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836817+0000", 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020725999999999999 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.836849+0000", 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2149e-05 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:08.937028+0000", 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100178188 2024-09-16T04:42:48.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.037317+0000", 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100289105 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.037342+0000", 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5812e-05 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.040305+0000", 2024-09-16T04:42:48.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002962059 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:09.040388+0000", 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3153999999999997e-05 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.892254762, 2024-09-16T04:42:48.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.396784737, 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:48.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.46e-07 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253567+0000", 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00085024700000000005 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253594+0000", 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6529e-05 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253616+0000", 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2373000000000001e-05 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448095+0000", 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19447869800000001 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448103+0000", 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9379999999999992e-06 2024-09-16T04:42:48.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649474+0000", 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137053299999999 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649502+0000", 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7973000000000001e-05 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.886556607999999, 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.391121679, 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258417+0000", 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.722e-06 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258658+0000", 2024-09-16T04:42:48.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024151700000000001 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258682+0000", 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3526000000000001e-05 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258695+0000", 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3225e-05 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548388+0000", 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28969289100000001 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548393+0000", 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2970000000000003e-06 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649523+0000", 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101130161 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649537+0000", 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3339999999999999e-05 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.884089967999998, 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38866693400000002, 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.200 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260883+0000", 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0750000000000001e-06 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260979+0000", 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6552999999999995e-05 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260998+0000", 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.261010+0000", 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1817999999999999e-05 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548799+0000", 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28778851599999999 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548839+0000", 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0371999999999999e-05 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649539+0000", 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100699682 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649549+0000", 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8050000000000001e-06 2024-09-16T04:42:48.204 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.882451670000002, 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38775422500000001, 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.205 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262521+0000", 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.009e-06 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262542+0000", 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0981000000000001e-05 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262560+0000", 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8224e-05 2024-09-16T04:42:48.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262572+0000", 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2155e-05 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649287+0000", 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38671449499999999 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649295+0000", 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8329999999999994e-06 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650254+0000", 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00095943799999999996 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650274+0000", 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0089999999999999e-05 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpRead(3.17s2 65/53 ECSubRead(tid=26, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.344820+0000", 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.800151245999999, 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20354055099999999, 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344820+0000", 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344820+0000", 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344823+0000", 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2740000000000002e-06 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344842+0000", 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8936e-05 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344880+0000", 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8801999999999999e-05 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.344900+0000", 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0066000000000001e-05 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548281+0000", 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20338055399999999 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548361+0000", 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9919000000000005e-05 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.359637+0000", 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.785334303999999, 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28247715000000001, 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359637+0000", 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359637+0000", 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359640+0000", 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5730000000000002e-06 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359648+0000", 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0069999999999997e-06 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359714+0000", 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6128000000000001e-05 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.359735+0000", 2024-09-16T04:42:48.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0579999999999999e-05 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642075+0000", 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28234038299999997 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642114+0000", 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9478999999999999e-05 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.366437+0000", 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.778534702000002, 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27593029899999999, 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366437+0000", 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366437+0000", 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366439+0000", 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9740000000000001e-06 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366448+0000", 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6229999999999992e-06 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366476+0000", 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7966e-05 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.366498+0000", 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1246e-05 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642331+0000", 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.275833571 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.642367+0000", 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5918999999999999e-05 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.221 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.721931304000002, 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.32087775200000002, 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423042+0000", 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.77e-06 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423051+0000", 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8100000000000004e-06 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423081+0000", 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0626000000000001e-05 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523175+0000", 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100093245 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742603+0000", 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21942863300000001 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.743918+0000", 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001314668 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1s4 70/67 ECSubWrite(tid=154, reqid=osd.4.0:151, at_version=0'0, trim_to=0'0, roll_forward_to=70'33))", 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.523286+0000", 2024-09-16T04:42:48.225 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 27.621685287999998, 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.22189425900000001, 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523286+0000", 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523286+0000", 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523288+0000", 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2529999999999999e-06 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523305+0000", 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7286e-05 2024-09-16T04:42:48.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523331+0000", 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6389000000000001e-05 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523352+0000", 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0619999999999999e-05 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742851+0000", 2024-09-16T04:42:48.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21949913700000001 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742882+0000", 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1092e-05 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.745111+0000", 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002228561 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.745181+0000", 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9920999999999997e-05 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 79/67 ECSubWrite(tid=188, reqid=client.4728.0:711, at_version=79'67, trim_to=0'0, roll_forward_to=79'65))", 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.714432+0000", 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 18.430539976999999, 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.303689612, 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.230 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714432+0000", 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714432+0000", 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714434+0000", 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.95e-06 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714485+0000", 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1632999999999998e-05 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.714523+0000", 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7954999999999998e-05 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814634+0000", 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100110872 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915010+0000", 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100376302 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915028+0000", 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7558999999999999e-05 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018009+0000", 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102981302 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018121+0000", 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112039 2024-09-16T04:42:48.234 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs3 79/67 ECSubWrite(tid=189, reqid=client.4728.0:712, at_version=79'68, trim_to=0'0, roll_forward_to=79'65))", 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.814716+0000", 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 18.330255147999999, 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20348661600000001, 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814716+0000", 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814716+0000", 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814718+0000", 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814775+0000", 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7293999999999997e-05 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.814807+0000", 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1894999999999999e-05 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.914910+0000", 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100103152 2024-09-16T04:42:48.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915929+0000", 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010187760000000001 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.915960+0000", 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0375999999999999e-05 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018136+0000", 2024-09-16T04:42:48.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10217612600000001 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.018203+0000", 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7306999999999998e-05 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 92/67 ECSubWrite(tid=240, reqid=osd.11.0:240, at_version=92'90, trim_to=0'0, roll_forward_to=90'88))", 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:42.923587+0000", 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.2213849139999997, 2024-09-16T04:42:48.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.208100591, 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.923587+0000", 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.923587+0000", 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.923588+0000", 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4300000000000001e-07 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.923645+0000", 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6975999999999998e-05 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.923671+0000", 2024-09-16T04:42:48.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6310000000000001e-05 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:43.023773+0000", 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100102449 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:43.023937+0000", 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016389200000000001 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:43.023964+0000", 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7019999999999999e-05 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:43.131648+0000", 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10768420200000001 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:43.131687+0000", 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8798999999999999e-05 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.244 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.244 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.244 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:48.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:42:48.246 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:48.242+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:48.268 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:48.346 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:48.343+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:48.375 INFO:tasks.rados.rados.0.smithi165.stdout:360: finishing write tid 5 to smithi16530038-9 2024-09-16T04:42:48.376 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:48.371+0000 7f4983483640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 18 after trimming was started 2024-09-16T04:42:48.421 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.ds2 97/67 ECSubWrite(tid=215, reqid=osd.7.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=97'197))", 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:48.371033+0000", 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.049596781, 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.049639118000000003, 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.422 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "no flag points reached", 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.371033+0000", 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.423 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.371033+0000", 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.371035+0000", 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.916e-06 2024-09-16T04:42:48.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.425 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.425 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.371057+0000", 2024-09-16T04:42:48.425 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.173e-05 2024-09-16T04:42:48.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.371086+0000", 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8932999999999999e-05 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpReadReply(3.19s0 97/58 ECSubReadReply(tid=143, attrs_read=0))", 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:48.379008+0000", 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.041621928000000002, 2024-09-16T04:42:48.426 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.041739230000000002, 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379008+0000", 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379008+0000", 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379010+0000", 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379045+0000", 2024-09-16T04:42:48.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4727000000000001e-05 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379071+0000", 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6543e-05 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379090+0000", 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8516e-05 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpReadReply(3.19s0 97/58 ECSubReadReply(tid=143, attrs_read=0))", 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:48.379081+0000", 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.041548532999999999, 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.041737521999999999, 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:48.430 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379081+0000", 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379081+0000", 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379082+0000", 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.64e-07 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379101+0000", 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9185999999999999e-05 2024-09-16T04:42:48.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379114+0000", 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2306999999999999e-05 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:48.379122+0000", 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3529999999999995e-06 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 3 2024-09-16T04:42:48.434 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:48.435 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:48.437 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:42:48.446 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:48.439+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:48.451 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:48.451 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:48.469 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:48.470 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:48.493 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:48.493 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:48.514 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:48.535 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:48.536 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:48.546 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:42:48.547 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:48.543+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:48.553 INFO:tasks.rados.rados.0.smithi165.stdout:362: done (1 left) 2024-09-16T04:42:48.553 INFO:tasks.rados.rados.0.smithi165.stdout:363: read oid 22 snap -1 2024-09-16T04:42:48.553 INFO:tasks.rados.rados.0.smithi165.stdout:363: expect deleted 2024-09-16T04:42:48.553 INFO:tasks.rados.rados.0.smithi165.stdout:364: copy_from oid 45 from oid 16 current snap is 30 2024-09-16T04:42:48.553 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 39 current snap is 30 2024-09-16T04:42:48.553 INFO:tasks.rados.rados.0.smithi165.stdout:365: seq_num 136 ranges {81920=737280,819200=770048,1589248=540672,2129920=245760} 2024-09-16T04:42:48.561 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:48.562 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:48.567 INFO:tasks.rados.rados.0.smithi165.stdout:365: writing smithi16530038-39 from 81920 to 819200 tid 1 2024-09-16T04:42:48.577 INFO:tasks.rados.rados.0.smithi165.stdout:365: writing smithi16530038-39 from 819200 to 1589248 tid 2 2024-09-16T04:42:48.584 INFO:tasks.rados.rados.0.smithi165.stdout:365: writing smithi16530038-39 from 1589248 to 2129920 tid 3 2024-09-16T04:42:48.587 INFO:tasks.rados.rados.0.smithi165.stdout:365: writing smithi16530038-39 from 2129920 to 2375680 tid 4 2024-09-16T04:42:48.587 INFO:tasks.rados.rados.0.smithi165.stdout:366: read oid 41 snap -1 2024-09-16T04:42:48.587 INFO:tasks.rados.rados.0.smithi165.stdout:366: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-16T04:42:48.587 INFO:tasks.rados.rados.0.smithi165.stdout:367: snap_create 2024-09-16T04:42:48.610 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:48.610 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:48.610 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:48.610 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:48.610 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:48.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:42:48.661 INFO:tasks.rados.rados.0.smithi165.stdout:360: finishing write tid 6 to smithi16530038-9 2024-09-16T04:42:48.662 INFO:tasks.rados.rados.0.smithi165.stdout:360: oid 9 updating version 0 to 93 2024-09-16T04:42:48.662 INFO:tasks.rados.rados.0.smithi165.stdout:360: oid 9 version 93 is already newer than 91 2024-09-16T04:42:48.662 INFO:tasks.rados.rados.0.smithi165.stdout:360: oid 9 version 93 is already newer than 92 2024-09-16T04:42:48.662 INFO:tasks.rados.rados.0.smithi165.stdout:360: oid 9 version 93 is already newer than 89 2024-09-16T04:42:48.662 INFO:tasks.rados.rados.0.smithi165.stdout:360: oid 9 version 93 is already newer than 90 2024-09-16T04:42:48.662 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 93 (ObjNum 135 snap 29 seq_num 135) dirty exists 2024-09-16T04:42:48.662 INFO:tasks.rados.rados.0.smithi165.stdout:360: left oid 9 (ObjNum 135 snap 29 seq_num 135) 2024-09-16T04:42:48.670 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:48.667+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:48.686 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:48.686 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:48.715 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:48.715 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:48.745 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:48.745 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:48.749 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:48.743+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:48.750 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:157 3.9s0 3:9de467d5:::smithi16530038-27:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 62.264252695000003, 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51218579099999995, 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.751 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 157 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484008+0000", 2024-09-16T04:42:48.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484010+0000", 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0329999999999998e-06 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484272+0000", 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026256800000000002 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484296+0000", 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3509000000000001e-05 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484308+0000", 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2699e-05 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584144+0000", 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099835667000000003 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.584304+0000", 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016037700000000001 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.585781+0000", 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001476455 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589205+0000", 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0034239499999999998 2024-09-16T04:42:48.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996077+0000", 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40687225300000002 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.996193+0000", 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011628 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:158 3.9s0 3:9de467d5:::smithi16530038-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 62.263929167000001, 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51808159200000004, 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 158 2024-09-16T04:42:48.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484332+0000", 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484339+0000", 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8630000000000002e-06 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484355+0000", 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6473e-05 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484364+0000", 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8950000000000004e-06 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.586917+0000", 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102553035 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.587177+0000", 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026005499999999998 2024-09-16T04:42:48.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.588403+0000", 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001225152 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589911+0000", 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015086260000000001 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002165+0000", 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41225355800000002 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002413+0000", 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024783200000000002 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.9s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:48.762 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 62.263860358000002, 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.62506688499999996, 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:48.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484401+0000", 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484405+0000", 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.827e-06 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484416+0000", 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1609e-05 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484425+0000", 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790928+0000", 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30650327300000002 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.791067+0000", 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013860200000000001 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003346+0000", 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212279577 2024-09-16T04:42:48.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003611+0000", 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026456899999999998 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109440+0000", 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105828857 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109467+0000", 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6900999999999999e-05 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.19s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 61.640907751, 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.976424197, 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:48.768 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:48.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107354+0000", 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9829999999999999e-06 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107362+0000", 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6129999999999998e-06 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107388+0000", 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5675000000000001e-05 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207482+0000", 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009412299999999 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207644+0000", 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000161741 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207649+0000", 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0359999999999997e-06 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082558+0000", 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8749096240000001 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082784+0000", 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022614900000000001 2024-09-16T04:42:48.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.083777+0000", 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099225300000000002 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:178 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 60.670146324999997, 2024-09-16T04:42:48.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1069151189999999, 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 178 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078116+0000", 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078127+0000", 2024-09-16T04:42:48.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.062e-05 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078151+0000", 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4304999999999999e-05 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078175+0000", 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4006e-05 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776496+0000", 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69832100500000005 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776503+0000", 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.533e-06 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185014+0000", 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40851148700000001 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185029+0000", 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4780000000000001e-05 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:179 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 60.670050963000001, 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1068670979999999, 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.778 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 179 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.779 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078211+0000", 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.44e-06 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078220+0000", 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078244+0000", 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3867999999999999e-05 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078260+0000", 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6203000000000001e-05 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780436+0000", 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70217526500000005 2024-09-16T04:42:48.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780441+0000", 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4620000000000004e-06 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185067+0000", 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40462590799999998 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185076+0000", 2024-09-16T04:42:48.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3219999999999999e-06 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:180 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 60.669977907000003, 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.106828843, 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.783 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 180 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078284+0000", 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078291+0000", 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.657e-06 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.785 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078310+0000", 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078325+0000", 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4772e-05 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981190+0000", 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90286589500000003 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981194+0000", 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0790000000000002e-06 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185100+0000", 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20390586399999999 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185111+0000", 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0794e-05 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:185 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 60.649661928, 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98955797199999995, 2024-09-16T04:42:48.788 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098600+0000", 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098611+0000", 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0423e-05 2024-09-16T04:42:48.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098642+0000", 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1532000000000002e-05 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098658+0000", 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5996000000000001e-05 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683726+0000", 2024-09-16T04:42:48.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58506727400000003 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683731+0000", 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088142+0000", 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40441113000000001 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088156+0000", 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4148000000000001e-05 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:186 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 60.649573873000001, 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.089963279, 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.793 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 186 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.794 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098687+0000", 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.063e-06 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098693+0000", 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6230000000000002e-06 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098705+0000", 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2019e-05 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198804+0000", 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098513 2024-09-16T04:42:48.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683799+0000", 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48499579100000001 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683801+0000", 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0889999999999998e-06 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188641+0000", 2024-09-16T04:42:48.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50483997300000005 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188650+0000", 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.208e-06 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:187 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 60.549424469999998, 2024-09-16T04:42:48.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98988477799999997, 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 187 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198838+0000", 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.86e-06 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198846+0000", 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0059999999999992e-06 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198864+0000", 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8468999999999999e-05 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198876+0000", 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1443e-05 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783957+0000", 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58508126900000001 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783962+0000", 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7230000000000001e-06 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188710+0000", 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40474714899999997 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188721+0000", 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0859000000000001e-05 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:583 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.413938514000002, 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2114807170000002, 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.803 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 583 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:48.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334325+0000", 2024-09-16T04:42:48.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6969999999999999e-06 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334340+0000", 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4508999999999999e-05 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334388+0000", 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8016000000000002e-05 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.806 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334413+0000", 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5284999999999999e-05 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828480+0000", 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406639099999999 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828485+0000", 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3430000000000003e-06 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.769267+0000", 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.94078222199999995 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.771858+0000", 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025913479999999998 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174292+0000", 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40243307299999997 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174565+0000", 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027297299999999998 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.176259+0000", 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001694161 2024-09-16T04:42:48.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281556+0000", 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10529764799999999 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545252+0000", 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.263695658 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545802+0000", 2024-09-16T04:42:48.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00055039300000000004 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:584 3.1as0 3:5f4281cc:::smithi16530038-45:head [rollback 11] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.413804051, 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2123693160000002, 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.811 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 584 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:48.812 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334458+0000", 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.663e-06 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334465+0000", 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2239999999999998e-06 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334485+0000", 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9925999999999999e-05 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334500+0000", 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5449000000000001e-05 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828570+0000", 2024-09-16T04:42:48.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49407003700000002 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828576+0000", 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.93e-06 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872096+0000", 2024-09-16T04:42:48.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.043520019 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872408+0000", 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031130200000000001 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177318+0000", 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30491066900000002 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.816 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177579+0000", 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026102699999999999 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.178857+0000", 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001277395 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281984+0000", 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10312769099999999 2024-09-16T04:42:48.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546082+0000", 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26409753200000002 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546825+0000", 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00074345200000000002 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:585 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.413740179000001, 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3685253789999998, 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 585 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334521+0000", 2024-09-16T04:42:48.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4169999999999999e-06 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334529+0000", 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8560000000000007e-06 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334547+0000", 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7813e-05 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334561+0000", 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3803e-05 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828622+0000", 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406153000000003 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828624+0000", 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.277e-06 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873077+0000", 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.044453106 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873349+0000", 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027208400000000002 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279758+0000", 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40640956900000003 2024-09-16T04:42:48.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279965+0000", 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000206028 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281051+0000", 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010864399999999999 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.282237+0000", 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011862839999999999 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.701843+0000", 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41960623800000002 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.703045+0000", 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001201934 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.825 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:635 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~524288] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.574880570000001, 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71243645600000005, 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 635 2024-09-16T04:42:48.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173382+0000", 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174349+0000", 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00096718499999999996 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174408+0000", 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9036000000000002e-05 2024-09-16T04:42:48.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174432+0000", 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4176e-05 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579526+0000", 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40509398299999999 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579719+0000", 2024-09-16T04:42:48.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019240699999999999 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.581395+0000", 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016763769999999999 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.586083+0000", 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0046879950000000004 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885565+0000", 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29948223000000002 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885816+0000", 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025068399999999999 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:799 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~524288] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:48.831 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.2552935200000004, 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.61375249600000004, 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 799 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492970+0000", 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.027e-06 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.493969+0000", 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099961899999999998 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494004+0000", 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5166000000000002e-05 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494056+0000", 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1807000000000001e-05 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494238+0000", 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018187800000000001 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494252+0000", 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.349e-05 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.593780+0000", 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099528025000000006 2024-09-16T04:42:48.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.594015+0000", 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023556099999999999 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.596480+0000", 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024650140000000002 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803670+0000", 2024-09-16T04:42:48.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.207189436 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.106478+0000", 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280800499999999 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.106719+0000", 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000241468 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.837 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:800 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~573440] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.2508259400000004, 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70919317999999998, 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.838 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 800 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.839 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497437+0000", 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.796e-06 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498329+0000", 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000892124 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498363+0000", 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3908999999999997e-05 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498379+0000", 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6124e-05 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498519+0000", 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000140245 2024-09-16T04:42:48.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498535+0000", 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.579e-05 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.599170+0000", 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100634526 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.842 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.599348+0000", 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017774199999999999 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.600830+0000", 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001482178 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803675+0000", 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20284549599999999 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206492+0000", 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40281652400000001 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206627+0000", 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013572599999999999 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:801 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~688128] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:48.844 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.2456238929999994, 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70428581800000001, 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 801 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502639+0000", 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503645+0000", 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001006002 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503676+0000", 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1526999999999998e-05 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503692+0000", 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5962999999999999e-05 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.847 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503829+0000", 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000137229 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503845+0000", 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5270000000000001e-05 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.602063+0000", 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098218277000000007 2024-09-16T04:42:48.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.602224+0000", 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000160996 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.603888+0000", 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016640330000000001 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803679+0000", 2024-09-16T04:42:48.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19979068899999999 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206810+0000", 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40313103500000003 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206922+0000", 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011254 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:802 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~294912] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.2434934549999994, 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70333736700000005, 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 802 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.851 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504768+0000", 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.48e-06 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505226+0000", 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00045742599999999998 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505255+0000", 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9810000000000001e-05 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.853 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505276+0000", 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0639999999999999e-05 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505379+0000", 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000103169 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505389+0000", 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8700000000000004e-06 2024-09-16T04:42:48.854 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.798131+0000", 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29274190900000002 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.798322+0000", 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000190912 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.799632+0000", 2024-09-16T04:42:48.855 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001310173 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803863+0000", 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0042312249999999999 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.207875+0000", 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40401177799999999 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.856 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.208104+0000", 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000228975 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:803 3.1as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (57) in=7b] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.2429490429999994, 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.010178128, 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:48.857 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 803 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.858 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505313+0000", 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.533e-06 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505320+0000", 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6930000000000007e-06 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.859 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505340+0000", 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.003e-05 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605448+0000", 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010718 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.210631+0000", 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60518353999999996 2024-09-16T04:42:48.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.210911+0000", 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027993799999999998 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.212558+0000", 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016472839999999999 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.315431+0000", 2024-09-16T04:42:48.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102872511 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.515321+0000", 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19989022000000001 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.515489+0000", 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016819900000000001 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.862 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:804 3.1as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.142777487, 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3152593989999999, 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 804 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.863 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:42:48.864 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605485+0000", 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9230000000000001e-06 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605493+0000", 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4069999999999993e-06 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:48.865 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605516+0000", 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2506000000000001e-05 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605528+0000", 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2734e-05 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.313900+0000", 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70837135200000001 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.866 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.314072+0000", 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017190200000000001 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.516098+0000", 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20202645 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.516435+0000", 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000336535 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:48.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.920742+0000", 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40430758999999999 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:48.868 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:48.869 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:48.870 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:48.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:42:48.871 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:48.863+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:48.872 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:48.872 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:48.872 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:48.873 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:48.887 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:48.887 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:48.911 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:42:48.911 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:42:48.970 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:48.966+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:49.006 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:49.006 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:49.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:42:49.072 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:49.067+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:49.141 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:49.141 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:49.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:42:49.172 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:49.167+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:49.272 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:49.272 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:49.273 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:49.266+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:49.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:42:49.374 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:49.371+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:49.415 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:49.416 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:49.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:49.477 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:49.475+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:49.550 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:49.550 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:49.554 INFO:tasks.rados.rados.0.smithi165.stdout:360: done (5 left) 2024-09-16T04:42:49.554 INFO:tasks.rados.rados.0.smithi165.stdout:363: done (4 left) 2024-09-16T04:42:49.554 INFO:tasks.rados.rados.0.smithi165.stdout:367: done (3 left) 2024-09-16T04:42:49.554 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 11 current snap is 31 2024-09-16T04:42:49.554 INFO:tasks.rados.rados.0.smithi165.stdout:368: seq_num 137 ranges {1720320=753664,2473984=786432,3260416=655360,3915776=114688} 2024-09-16T04:42:49.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:49.575 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:49.570+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:49.578 INFO:tasks.rados.rados.0.smithi165.stdout:368: writing smithi16530038-11 from 1720320 to 2473984 tid 1 2024-09-16T04:42:49.584 INFO:tasks.rados.rados.0.smithi165.stdout:368: writing smithi16530038-11 from 2473984 to 3260416 tid 2 2024-09-16T04:42:49.589 INFO:tasks.rados.rados.0.smithi165.stdout:368: writing smithi16530038-11 from 3260416 to 3915776 tid 3 2024-09-16T04:42:49.590 INFO:tasks.rados.rados.0.smithi165.stdout:368: writing smithi16530038-11 from 3915776 to 4030464 tid 4 2024-09-16T04:42:49.590 INFO:tasks.rados.rados.0.smithi165.stdout:369: rollback oid 24 current snap is 31 2024-09-16T04:42:49.590 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 27 2024-09-16T04:42:49.590 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 8 current snap is 31 2024-09-16T04:42:49.590 INFO:tasks.rados.rados.0.smithi165.stdout:370: seq_num 138 ranges {0=770048,770048=540672,1310720=753664,2064384=262144} 2024-09-16T04:42:49.596 INFO:tasks.rados.rados.0.smithi165.stdout:370: writing smithi16530038-8 from 0 to 770048 tid 1 2024-09-16T04:42:49.600 INFO:tasks.rados.rados.0.smithi165.stdout:370: writing smithi16530038-8 from 770048 to 1310720 tid 2 2024-09-16T04:42:49.605 INFO:tasks.rados.rados.0.smithi165.stdout:370: writing smithi16530038-8 from 1310720 to 2064384 tid 3 2024-09-16T04:42:49.607 INFO:tasks.rados.rados.0.smithi165.stdout:370: writing smithi16530038-8 from 2064384 to 2326528 tid 4 2024-09-16T04:42:49.608 INFO:tasks.rados.rados.0.smithi165.stdout:371: rmattr oid 5 current snap is 31 2024-09-16T04:42:49.608 INFO:tasks.rados.rados.0.smithi165.stdout:371: done (6 left) 2024-09-16T04:42:49.608 INFO:tasks.rados.rados.0.smithi165.stdout:372: read oid 27 snap -1 2024-09-16T04:42:49.608 INFO:tasks.rados.rados.0.smithi165.stdout:372: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:49.608 INFO:tasks.rados.rados.0.smithi165.stdout:373: snap_create 2024-09-16T04:42:49.675 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:49.671+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:49.702 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:49.702 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:49.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:49.777 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:49.771+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:49.878 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:49.871+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:49.883 INFO:tasks.rados.rados.0.smithi165.stdout:372: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:49.889 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:49.889 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:49.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:49.979 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:42:49.975+0000 7f09754f8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26665) UID: 0 2024-09-16T04:42:49.979 INFO:tasks.rados.rados.0.smithi165.stdout:366: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-16T04:42:50.039 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:50.039 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:50.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:50.080 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:50.074+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:50.181 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:50.174+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:50.203 INFO:tasks.rados.rados.0.smithi165.stdout:365: finishing write tid 1 to smithi16530038-39 2024-09-16T04:42:50.205 INFO:tasks.rados.rados.0.smithi165.stdout:365: finishing write tid 2 to smithi16530038-39 2024-09-16T04:42:50.206 INFO:tasks.rados.rados.0.smithi165.stdout:365: finishing write tid 3 to smithi16530038-39 2024-09-16T04:42:50.208 INFO:tasks.rados.rados.0.smithi165.stdout:365: finishing write tid 4 to smithi16530038-39 2024-09-16T04:42:50.208 INFO:tasks.rados.rados.0.smithi165.stdout:365: finishing write tid 5 to smithi16530038-39 2024-09-16T04:42:50.222 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:50.222 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:50.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:42:50.246 INFO:tasks.rados.rados.0.smithi165.stdout:370: finishing write tid 1 to smithi16530038-8 2024-09-16T04:42:50.282 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:50.279+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:50.301 INFO:tasks.rados.rados.0.smithi165.stdout:369: finishing rollback tid 0 to smithi16530038-24 2024-09-16T04:42:50.301 INFO:tasks.rados.rados.0.smithi165.stdout:369: finishing rollback tid 1 to smithi16530038-24 2024-09-16T04:42:50.301 INFO:tasks.rados.rados.0.smithi165.stdout:369: finishing rollback tid 2 to smithi16530038-24 2024-09-16T04:42:50.301 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 46 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:42:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:365: finishing write tid 6 to smithi16530038-39 2024-09-16T04:42:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:365: oid 39 updating version 0 to 45 2024-09-16T04:42:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:365: oid 39 version 45 is already newer than 44 2024-09-16T04:42:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:365: oid 39 updating version 45 to 47 2024-09-16T04:42:50.315 INFO:tasks.rados.rados.0.smithi165.stdout:365: oid 39 version 47 is already newer than 46 2024-09-16T04:42:50.315 INFO:tasks.rados.rados.0.smithi165.stdout:365: oid 39 version 47 is already newer than 43 2024-09-16T04:42:50.315 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 47 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-09-16T04:42:50.315 INFO:tasks.rados.rados.0.smithi165.stdout:365: left oid 39 (ObjNum 136 snap 30 seq_num 136) 2024-09-16T04:42:50.355 INFO:tasks.rados.rados.0.smithi165.stdout:370: finishing write tid 2 to smithi16530038-8 2024-09-16T04:42:50.357 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:50.357 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:50.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T04:42:50.382 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:50.379+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:50.483 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:50.479+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:50.542 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:50.542 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:50.555 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:42:50.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:42:50.555 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-16T04:42:50.556 INFO:tasks.thrashosds.thrasher:Killing osd 15, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T04:42:50.556 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-16T04:42:50.556 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:42:50.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T04:42:50.567 INFO:tasks.rados.rados.0.smithi165.stdout:365: done (7 left) 2024-09-16T04:42:50.567 INFO:tasks.rados.rados.0.smithi165.stdout:366: done (6 left) 2024-09-16T04:42:50.567 INFO:tasks.rados.rados.0.smithi165.stdout:369: done (5 left) 2024-09-16T04:42:50.567 INFO:tasks.rados.rados.0.smithi165.stdout:372: done (4 left) 2024-09-16T04:42:50.567 INFO:tasks.rados.rados.0.smithi165.stdout:373: done (3 left) 2024-09-16T04:42:50.567 INFO:tasks.rados.rados.0.smithi165.stdout:374: delete oid 25 current snap is 32 2024-09-16T04:42:50.575 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:42:50.622 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:50.618+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:50.648 INFO:tasks.ceph.osd.15:Stopped 2024-09-16T04:42:50.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 15 2024-09-16T04:42:50.670 INFO:tasks.rados.rados.0.smithi165.stdout:370: finishing write tid 3 to smithi16530038-8 2024-09-16T04:42:50.673 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-16T04:42:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:42:50.676 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:50.684 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:50.679+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:50.785 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:50.778+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:50.817 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:50.818 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:50.832 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:50.886 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:50.883+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:50.905 INFO:tasks.rados.rados.0.smithi165.stdout:374: done (3 left) 2024-09-16T04:42:50.905 INFO:tasks.rados.rados.0.smithi165.stdout:375: rollback oid 34 current snap is 32 2024-09-16T04:42:50.906 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 31 2024-09-16T04:42:50.906 INFO:tasks.rados.rados.0.smithi165.stdout:376: snap_create 2024-09-16T04:42:50.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:42:50.960 INFO:tasks.rados.rados.0.smithi165.stdout:370: finishing write tid 4 to smithi16530038-8 2024-09-16T04:42:50.983 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-09-16T04:42:50.978+0000 7f2decafd640 -1 mon.a@0(leader).osd e100 definitely_dead 0 2024-09-16T04:42:50.986 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:50.982+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:50.993 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:50.994 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:51.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:51.088 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:51.083+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:51.113 INFO:tasks.rados.rados.0.smithi165.stdout:375: finishing rollback tid 0 to smithi16530038-34 2024-09-16T04:42:51.133 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:51.133 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:51.151 INFO:tasks.rados.rados.0.smithi165.stdout:364: finishing copy_from racing read to smithi16530038-45 2024-09-16T04:42:51.153 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:51.154 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:51.175 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s4 36/32 ECSubWrite(tid=8, reqid=client.4728.0:109, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 65.799130887000004, 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201785732, 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.176 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374506+0000", 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374507+0000", 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374649+0000", 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014185599999999999 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374672+0000", 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3277000000000001e-05 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.374685+0000", 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2797e-05 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574940+0000", 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20025501600000001 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574956+0000", 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5936000000000001e-05 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576258+0000", 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001302388 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.576291+0000", 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.180 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 36/32 ECSubWrite(tid=4, reqid=client.4728.0:112, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 65.600024559999994, 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20173359499999999, 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573612+0000", 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.573614+0000", 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.114e-06 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574099+0000", 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048458300000000002 2024-09-16T04:42:51.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574141+0000", 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2551e-05 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.574168+0000", 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6899000000000001e-05 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674350+0000", 2024-09-16T04:42:51.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100181464 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.674362+0000", 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2138e-05 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775305+0000", 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100943171 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:45.775345+0000", 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0674999999999999e-05 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:164 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 64.276102125999998, 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40676020400000001, 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.185 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 164 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:42:51.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897538+0000", 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0500000000000002e-06 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897554+0000", 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5886e-05 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897591+0000", 2024-09-16T04:42:51.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6899e-05 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897617+0000", 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5922e-05 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897727+0000", 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011011200000000001 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897835+0000", 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107889 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304263+0000", 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40642834999999999 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304295+0000", 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1096000000000002e-05 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:165 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 64.275962437999993, 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20425574899999999, 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 165 2024-09-16T04:42:51.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897676+0000", 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897685+0000", 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.183e-06 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897709+0000", 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4216e-05 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997840+0000", 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131417 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098015+0000", 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100174894 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098124+0000", 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108714 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101885+0000", 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037611459999999999 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101930+0000", 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4675999999999997e-05 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.194 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:166 3.6s0 3.a17ee16 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 64.175746175, 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.306539124, 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 166 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997892+0000", 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:42:51.196 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997905+0000", 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2583e-05 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997926+0000", 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.101e-05 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098022+0000", 2024-09-16T04:42:51.197 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100096478 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304408+0000", 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20638583099999999 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304429+0000", 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1132e-05 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.16s0 42/41 ECSubRead(tid=6, to_read={3:6877e850:::smithi16530038-5:head=0,20480,0}, subchunks={3:6877e850:::smithi16530038-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 58.865413109000002, 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20348143299999999, 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:51.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308260+0000", 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6318e-05 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308312+0000", 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2611999999999997e-05 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511485+0000", 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20317243800000001 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511705+0000", 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220065 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.202 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 58.864737276, 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20303517200000001, 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308901+0000", 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308970+0000", 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9240999999999997e-05 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309005+0000", 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4314999999999999e-05 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309023+0000", 2024-09-16T04:42:51.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8879999999999999e-05 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511826+0000", 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202802133 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511934+0000", 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108756 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 58.864502483999999, 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20302847199999999, 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309135+0000", 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309242+0000", 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106959 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309260+0000", 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309270+0000", 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0339e-05 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512036+0000", 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20276591199999999 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512162+0000", 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126675 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.14s4 43/41 ECSubWrite(tid=30, reqid=client.4728.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:42:51.210 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 57.834360082000003, 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20195774699999999, 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339277+0000", 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.61e-07 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339289+0000", 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1533e-05 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339303+0000", 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4002000000000001e-05 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439391+0000", 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10008851100000001 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439604+0000", 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301100000000001 2024-09-16T04:42:51.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439617+0000", 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3076e-05 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541138+0000", 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10152072299999999 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541234+0000", 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6130000000000003e-05 2024-09-16T04:42:51.214 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 53/32 ECSubWrite(tid=89, reqid=client.4728.0:426, at_version=53'29, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.239442064999999, 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20199651099999999, 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934195+0000", 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934336+0000", 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014033399999999999 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934357+0000", 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1769000000000001e-05 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034519+0000", 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016164599999999 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034685+0000", 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000166312 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.218 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034712+0000", 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6959999999999999e-05 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136138+0000", 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101425772 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136191+0000", 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2868000000000003e-05 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.219 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=115, reqid=client.4728.0:457, at_version=58'26, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.874280906000003, 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.240164188, 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:51.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5699999999999999e-07 2024-09-16T04:42:51.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299490+0000", 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013428299999999999 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299510+0000", 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437999999999999e-05 2024-09-16T04:42:51.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299523+0000", 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3132000000000001e-05 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436652+0000", 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13712892300000001 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436659+0000", 2024-09-16T04:42:51.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4410000000000004e-06 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539460+0000", 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102800512 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539520+0000", 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0201999999999997e-05 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=122, reqid=client.4728.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=58'29))", 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.361155095000001, 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20212734700000001, 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.225 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812484+0000", 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3700000000000002e-06 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812511+0000", 2024-09-16T04:42:51.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.775e-05 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812550+0000", 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812567+0000", 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7186000000000001e-05 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812770+0000", 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202862 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812799+0000", 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8716000000000001e-05 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014571+0000", 2024-09-16T04:42:51.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201771586 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014609+0000", 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8124999999999998e-05 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 35.920843327, 2024-09-16T04:42:51.229 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.296481516, 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252794+0000", 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253432+0000", 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063874400000000001 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253463+0000", 2024-09-16T04:42:51.231 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0052999999999998e-05 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253487+0000", 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4335999999999999e-05 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344896+0000", 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.091408780999999995 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344925+0000", 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9490000000000001e-05 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549182+0000", 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.204257051 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549275+0000", 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2464000000000001e-05 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.233 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 35.915052471000003, 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29077158600000003, 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258586+0000", 2024-09-16T04:42:51.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258848+0000", 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026226500000000002 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258888+0000", 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0395999999999997e-05 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258917+0000", 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8371999999999999e-05 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.346992+0000", 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.088075353999999995 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.347031+0000", 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549288+0000", 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20225760800000001 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549356+0000", 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7222000000000001e-05 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:51.238 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 35.912185737000001, 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.288032389, 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261453+0000", 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.79e-06 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261566+0000", 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000113801 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.240 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261597+0000", 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0772999999999998e-05 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261613+0000", 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348618+0000", 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.087004695000000007 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.241 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348634+0000", 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6246e-05 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549401+0000", 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200766994 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549483+0000", 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2150999999999993e-05 2024-09-16T04:42:51.242 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 35.910490717000002, 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28676506400000001, 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.243 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263148+0000", 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.81e-06 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.244 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263180+0000", 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2019e-05 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263210+0000", 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0429999999999998e-05 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263226+0000", 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6093e-05 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.245 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548818+0000", 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28559150300000002 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548834+0000", 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6804e-05 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549811+0000", 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00097657100000000008 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.246 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549911+0000", 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9834000000000007e-05 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/67 ECSubWrite(tid=159, reqid=client.4728.0:622, at_version=74'134, trim_to=0'0, roll_forward_to=72'132))", 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.700039990000001, 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.205798497, 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.247 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.248 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473597+0000", 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3999999999999998e-07 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473620+0000", 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3383999999999998e-05 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473630+0000", 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.249 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473639+0000", 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9799999999999993e-06 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676469+0000", 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202829168 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676503+0000", 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4239999999999997e-05 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.250 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679328+0000", 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028252860000000002 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679395+0000", 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6773000000000006e-05 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.251 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.21s5,QREQUEST,e:74,from:2(0),reservation_nonce:2)", 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.237071052000001, 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30070015300000003, 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.252 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936567+0000", 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.86e-06 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.253 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936576+0000", 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0559999999999994e-06 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936605+0000", 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8632999999999998e-05 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.036785+0000", 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10017992000000001 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237143+0000", 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20035802999999999 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237193+0000", 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0121999999999999e-05 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237266+0000", 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2532000000000003e-05 2024-09-16T04:42:51.255 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.bs1 76/67 ECSubRead(tid=164, to_read={3:d388eed4:::smithi16530038-28:head=0,339968,0}, subchunks={3:d388eed4:::smithi16530038-28:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.895729911, 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22646649499999999, 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.256 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277908+0000", 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7439999999999999e-06 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.257 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277925+0000", 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7084000000000001e-05 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277960+0000", 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4208999999999997e-05 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.258 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277978+0000", 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8219e-05 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504200+0000", 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22622223899999999 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504373+0000", 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000173 2024-09-16T04:42:51.259 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 90/37 ECSubWrite(tid=82, reqid=osd.15.0:82, at_version=90'74, trim_to=0'0, roll_forward_to=88'70))", 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:40.482601+0000", 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.691035626, 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20413171399999999, 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.260 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482601+0000", 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482601+0000", 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482603+0000", 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1629999999999999e-06 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.261 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482659+0000", 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6325e-05 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482690+0000", 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0442000000000001e-05 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482713+0000", 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3238999999999999e-05 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.262 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.584033+0000", 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101320049 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.584054+0000", 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0679e-05 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.686637+0000", 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10258316100000001 2024-09-16T04:42:51.263 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.686733+0000", 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5655999999999994e-05 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.264 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:51.266 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:51.266 INFO:tasks.rados.rados.0.smithi165.stdout:375: finishing rollback tid 1 to smithi16530038-34 2024-09-16T04:42:51.266 INFO:tasks.rados.rados.0.smithi165.stdout:364: finishing copy_from to smithi16530038-45 2024-09-16T04:42:51.266 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 63 (ObjNum 112 snap 21 seq_num 112) dirty exists 2024-09-16T04:42:51.266 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:51.266 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:51.263+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:51.267 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:51.268 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:51.268 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:51.269 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:51.269 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:51.269 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:51.287 INFO:tasks.rados.rados.0.smithi165.stdout:370: finishing write tid 5 to smithi16530038-8 2024-09-16T04:42:51.318 INFO:tasks.rados.rados.0.smithi165.stdout:375: finishing rollback tid 2 to smithi16530038-34 2024-09-16T04:42:51.318 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 99 (ObjNum 108 snap 20 seq_num 108) dirty exists 2024-09-16T04:42:51.367 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:51.363+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:51.381 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:51.381 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:51.401 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:51.401 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:51.401 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:51.401 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:51.408 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:51.409 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:51.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:51.434 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:51.434 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:51.467 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:51.463+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:51.486 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:51.486 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:51.507 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:51.529 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:51.530 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:51.544 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:51.544 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:51.544 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:51.545 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:51.545 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:51.547 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:51.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:51.572 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:51.573 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:51.566+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:42:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:marked down osd.15. 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:364: done (4 left) 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:375: done (3 left) 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:376: done (2 left) 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:377: rollback oid 46 current snap is 33 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 28 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:378: rollback oid 1 current snap is 33 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 29 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:379: rmattr oid 13 current snap is 33 2024-09-16T04:42:51.581 INFO:tasks.rados.rados.0.smithi165.stdout:379: done (4 left) 2024-09-16T04:42:51.582 INFO:tasks.rados.rados.0.smithi165.stdout:380: copy_from oid 14 from oid 49 current snap is 33 2024-09-16T04:42:51.582 INFO:tasks.rados.rados.0.smithi165.stdout:381: snap_create 2024-09-16T04:42:51.602 INFO:tasks.rados.rados.0.smithi165.stdout:370: finishing write tid 6 to smithi16530038-8 2024-09-16T04:42:51.603 INFO:tasks.rados.rados.0.smithi165.stdout:370: oid 8 updating version 0 to 90 2024-09-16T04:42:51.603 INFO:tasks.rados.rados.0.smithi165.stdout:370: oid 8 version 90 is already newer than 89 2024-09-16T04:42:51.603 INFO:tasks.rados.rados.0.smithi165.stdout:370: oid 8 updating version 90 to 91 2024-09-16T04:42:51.603 INFO:tasks.rados.rados.0.smithi165.stdout:370: oid 8 version 91 is already newer than 88 2024-09-16T04:42:51.603 INFO:tasks.rados.rados.0.smithi165.stdout:370: oid 8 updating version 91 to 92 2024-09-16T04:42:51.603 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 92 (ObjNum 138 snap 31 seq_num 138) dirty exists 2024-09-16T04:42:51.603 INFO:tasks.rados.rados.0.smithi165.stdout:370: left oid 8 (ObjNum 138 snap 31 seq_num 138) 2024-09-16T04:42:51.605 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:51.606 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:51.609 INFO:tasks.thrashosds.thrasher:Removing osd 15, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-16T04:42:51.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 15 2024-09-16T04:42:51.669 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:51.663+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:51.700 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:137 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 65.636072859999999, 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50711364199999998, 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.701 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 137 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062314+0000", 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062318+0000", 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1710000000000004e-06 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.703 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062327+0000", 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0089999999999996e-06 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162444+0000", 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011658 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163173+0000", 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072917800000000003 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163440+0000", 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026697899999999998 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164004+0000", 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056357499999999997 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164775+0000", 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00077136499999999998 2024-09-16T04:42:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569249+0000", 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.404473996 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569427+0000", 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177906 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:138 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 65.535903977000004, 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.71018989099999996, 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 138 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.707 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162484+0000", 2024-09-16T04:42:51.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162491+0000", 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162507+0000", 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5141e-05 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162514+0000", 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8460000000000008e-06 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165370+0000", 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028555719999999998 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165438+0000", 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8466999999999993e-05 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570190+0000", 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40475178899999997 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570271+0000", 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0437000000000003e-05 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.872672+0000", 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30240151100000001 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:181 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 63.606030711000002, 2024-09-16T04:42:51.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.514197601, 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 181 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092358+0000", 2024-09-16T04:42:51.714 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9500000000000001e-06 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093295+0000", 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00093678900000000005 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093330+0000", 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5039000000000001e-05 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093348+0000", 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7210999999999998e-05 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269509+0000", 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17616195900000001 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269804+0000", 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029419599999999999 2024-09-16T04:42:51.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272744+0000", 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029401789999999998 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279866+0000", 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007122499 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606426+0000", 2024-09-16T04:42:51.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32655929 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606553+0000", 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000127489 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:182 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 63.599702266000001, 2024-09-16T04:42:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50813697599999996, 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 182 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:51.719 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0260000000000002e-06 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099161+0000", 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000474556 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099192+0000", 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1612000000000002e-05 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099207+0000", 2024-09-16T04:42:51.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4796e-05 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275225+0000", 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17601765799999999 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275471+0000", 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024600099999999998 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001437422 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280012+0000", 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031043899999999998 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606726+0000", 2024-09-16T04:42:51.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32671404399999998 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606821+0000", 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4470999999999997e-05 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:183 3.bs0 3:d388eed4:::smithi16530038-28:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:51.724 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 63.599151857000003, 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60643071599999998, 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 183 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.725 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099235+0000", 2024-09-16T04:42:51.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.017e-06 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099242+0000", 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4230000000000002e-06 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099256+0000", 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3936e-05 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099264+0000", 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278076+0000", 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17881208600000001 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278318+0000", 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024238100000000001 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279368+0000", 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010492209999999999 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281553+0000", 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021853739999999999 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705572+0000", 2024-09-16T04:42:51.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.42401889100000001 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705665+0000", 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3173999999999999e-05 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:191 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:51.730 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 63.597291702, 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68460226999999996, 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 191 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101096+0000", 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-06 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101104+0000", 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1240000000000005e-06 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101122+0000", 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8045000000000001e-05 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101130+0000", 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2670000000000006e-06 2024-09-16T04:42:51.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280859+0000", 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179728363 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280864+0000", 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7370000000000003e-06 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.405994+0000", 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.12512986500000001 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.406243+0000", 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024852700000000001 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.785697+0000", 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.37945393900000002 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.735 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:192 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 63.597239483999999, 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68518017200000003, 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.736 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 192 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101154+0000", 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3300000000000004e-06 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101160+0000", 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.348e-06 2024-09-16T04:42:51.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101164+0000", 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2899999999999996e-06 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281071+0000", 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179906749 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:51.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281074+0000", 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.976e-06 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506776+0000", 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22570168299999999 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506893+0000", 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011752300000000001 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786327+0000", 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27943377000000003 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:201 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:51.741 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 63.597106396999997, 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68324074999999995, 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.744 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 201 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101282+0000", 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9750000000000001e-06 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101287+0000", 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.651e-06 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.201419+0000", 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10013264099999999 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482351+0000", 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28093166899999999 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482482+0000", 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013094199999999999 2024-09-16T04:42:51.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.784521+0000", 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30203853899999999 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:372 3.1s0 3:8583b9f2:::smithi16530038-39:3 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 52.517104099000001, 2024-09-16T04:42:51.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.58438020099999999, 2024-09-16T04:42:51.749 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.749 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.749 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.749 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.749 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.749 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 372 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181283+0000", 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1100000000000005e-07 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181288+0000", 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4739999999999996e-06 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181300+0000", 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1266e-05 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181308+0000", 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5669999999999995e-06 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181619+0000", 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031025600000000002 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181627+0000", 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3329999999999997e-06 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288322+0000", 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669511499999999 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288524+0000", 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202511 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.765662+0000", 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.47713786800000002 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:452 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.439432242999999, 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57389822599999996, 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 452 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.755 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258957+0000", 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9510000000000002e-06 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260061+0000", 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011037410000000001 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260099+0000", 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8401e-05 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260123+0000", 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3808999999999999e-05 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260226+0000", 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010314000000000001 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260310+0000", 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3652000000000005e-05 2024-09-16T04:42:51.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.261217+0000", 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090691199999999995 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.262686+0000", 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014689 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832721+0000", 2024-09-16T04:42:51.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57003547600000004 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832852+0000", 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013124400000000001 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:453 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~622592] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.428754369000004, 2024-09-16T04:42:51.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56351327799999995, 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 453 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:51.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269634+0000", 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2069999999999998e-06 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270168+0000", 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053375500000000002 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270185+0000", 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7231000000000002e-05 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270194+0000", 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4349999999999997e-06 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270266+0000", 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2702999999999998e-05 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270366+0000", 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9485e-05 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271233+0000", 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00086726900000000005 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374033+0000", 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10280007300000001 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833038+0000", 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45900526899999999 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833145+0000", 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106851 2024-09-16T04:42:51.765 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:454 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~147456] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.426489588999999, 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56224230100000006, 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.766 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 454 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271898+0000", 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.364e-06 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272075+0000", 2024-09-16T04:42:51.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017664100000000001 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272088+0000", 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3192e-05 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272097+0000", 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4720000000000001e-06 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372245+0000", 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10014781 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372332+0000", 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7258999999999999e-05 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372854+0000", 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052174600000000002 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374728+0000", 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018736219999999999 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834034+0000", 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45930626099999999 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834139+0000", 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000104934 2024-09-16T04:42:51.771 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:455 3.1s0 3:803446a7:::smithi16530038-31:head [setxattr _header (57) in=7b] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.426265225999998, 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.66155386500000002, 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.772 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 455 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272122+0000", 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272127+0000", 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5369999999999997e-06 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272135+0000", 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:42:51.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272141+0000", 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5999999999999997e-06 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373376+0000", 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101234779 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373445+0000", 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9507999999999999e-05 2024-09-16T04:42:51.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373742+0000", 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029695500000000001 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374731+0000", 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098912400000000008 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933530+0000", 2024-09-16T04:42:51.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.55879828600000003 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933675+0000", 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145353 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:456 3.1s0 3:803446a7:::smithi16530038-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:51.777 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.426234043000001, 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.76406687699999998, 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 456 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272153+0000", 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0100000000000005e-07 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272155+0000", 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.638e-06 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272165+0000", 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1579999999999994e-06 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272171+0000", 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:42:51.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373963+0000", 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10179178799999999 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374024+0000", 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0791999999999997e-05 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934462+0000", 2024-09-16T04:42:51.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56043857600000002 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934706+0000", 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024387000000000001 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.036219+0000", 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10151300000000001 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.782 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:458 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.394964979000001, 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50818626099999997, 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.783 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 458 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:42:51.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303426+0000", 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5870000000000003e-06 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303961+0000", 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053495599999999995 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303975+0000", 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3835999999999999e-05 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404043+0000", 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100068845 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404113+0000", 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9350999999999994e-05 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404242+0000", 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012877400000000001 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405312+0000", 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001070812 2024-09-16T04:42:51.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407819+0000", 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002506131 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811485+0000", 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40366653800000002 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811608+0000", 2024-09-16T04:42:51.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012243099999999999 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:461 3.bs0 3:d1897791:::smithi16530038-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.294232276999999, 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.61182177000000004, 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.789 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 461 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:51.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6399999999999998e-07 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404155+0000", 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2550000000000001e-06 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404160+0000", 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0890000000000002e-06 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404162+0000", 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8399999999999999e-06 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507951+0000", 2024-09-16T04:42:51.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103788373 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.508127+0000", 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017614900000000001 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813061+0000", 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30493453100000001 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813333+0000", 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027161699999999998 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.015976+0000", 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20264265200000001 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:574 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:42:51.794 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.394623289999998, 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50935247100000003, 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 574 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:42:51.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303765+0000", 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.827e-06 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.796 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303772+0000", 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6210000000000004e-06 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303788+0000", 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5760000000000002e-05 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.403903+0000", 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100114464 2024-09-16T04:42:51.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404144+0000", 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024159600000000001 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404641+0000", 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000496598 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.406835+0000", 2024-09-16T04:42:51.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021939149999999998 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511811+0000", 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.104976429 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.812951+0000", 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30113980800000001 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.813115+0000", 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016445300000000001 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:576 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.294357625000004, 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51026414200000003, 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.800 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 576 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.801 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6199999999999998e-07 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404032+0000", 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6280000000000001e-06 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404038+0000", 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6200000000000001e-06 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404043+0000", 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5020000000000003e-06 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.510735+0000", 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669203400000001 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.510837+0000", 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101847 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511366+0000", 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052975600000000004 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.512713+0000", 2024-09-16T04:42:51.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001346168 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.914111+0000", 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40139793200000001 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.914293+0000", 2024-09-16T04:42:51.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018219300000000001 2024-09-16T04:42:51.806 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.806 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.806 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:644 3.bs0 3:d388eed4:::smithi16530038-28:head [read 0~1359872,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.521075584999998, 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51132577199999996, 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:51.807 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 644 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7e-07 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177315+0000", 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.523e-06 2024-09-16T04:42:51.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177322+0000", 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.968e-06 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177327+0000", 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.631e-06 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.458181+0000", 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28085389999999999 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.458352+0000", 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017090399999999999 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.688636+0000", 2024-09-16T04:42:51.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.23028437600000001 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:717 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~802816] snapc 16=[16,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.250091131000001, 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50523997499999995, 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:51.812 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 717 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448297+0000", 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2539999999999999e-06 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449278+0000", 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098078800000000002 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449311+0000", 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2768999999999997e-05 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449326+0000", 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5392000000000001e-05 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.537454+0000", 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.088127592000000005 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.537466+0000", 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2189e-05 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.544288+0000", 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0068214950000000003 2024-09-16T04:42:51.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.544574+0000", 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000286113 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.547183+0000", 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026089809999999998 2024-09-16T04:42:51.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.651536+0000", 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.104353801 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.953403+0000", 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30186636999999999 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.953535+0000", 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013223099999999999 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.819 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:51.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:51.821 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:51.815+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:51.825 INFO:tasks.rados.rados.0.smithi165.stdout:378: finishing rollback tid 0 to smithi16530038-1 2024-09-16T04:42:51.826 INFO:tasks.rados.rados.0.smithi165.stdout:378: finishing rollback tid 1 to smithi16530038-1 2024-09-16T04:42:51.922 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:51.919+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:51.926 INFO:tasks.rados.rados.0.smithi165.stdout:378: finishing rollback tid 2 to smithi16530038-1 2024-09-16T04:42:51.926 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 10 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:51.973 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-09-16T04:42:51.966+0000 7f2decafd640 -1 mon.a@0(leader).osd e101 definitely_dead 0 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:943 3.5s1 3.2115b2a5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:51.580647+0000", 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.40408632900000002, 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40412889099999999, 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.986 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "waiting for peered", 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 943 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580647+0000", 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.987 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580647+0000", 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580651+0000", 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9249999999999997e-06 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580668+0000", 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6949999999999999e-05 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580699+0000", 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1359999999999998e-05 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580716+0000", 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6905000000000002e-05 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580807+0000", 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0600999999999995e-05 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0634e-05 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.584336+0000", 2024-09-16T04:42:51.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003518916 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.584341+0000", 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8919999999999999e-06 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:944 3.5s1 3.2115b2a5 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:51.580752+0000", 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.40398090599999997, 2024-09-16T04:42:51.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.404169004, 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "waiting for peered", 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 944 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580752+0000", 2024-09-16T04:42:51.992 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580752+0000", 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:51.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580753+0000", 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2200000000000003e-07 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580757+0000", 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9519999999999996e-06 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580768+0000", 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1189000000000001e-05 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580776+0000", 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6729999999999992e-06 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580919+0000", 2024-09-16T04:42:51.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014272300000000001 2024-09-16T04:42:51.996 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.996 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.996 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:51.996 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580927+0000", 2024-09-16T04:42:51.996 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4440000000000008e-06 2024-09-16T04:42:51.996 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.684488+0000", 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103560847 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.684493+0000", 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.904e-06 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:51.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDPGPush(3.1ds5 101/98 [PushOp(3:bc30766e:::smithi16530038-11:head, version: 72'112, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bc30766e:::smithi16530038-11:head@72'112, size: 1720320, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:51.931033+0000", 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.053700316999999997, 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.054079097, 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:51.998 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.931033+0000", 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.931033+0000", 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:51.999 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.931035+0000", 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8029999999999999e-06 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.931505+0000", 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047036700000000002 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.000 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.931550+0000", 2024-09-16T04:42:52.001 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4948999999999999e-05 2024-09-16T04:42:52.001 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.001 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.001 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.001 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.931581+0000", 2024-09-16T04:42:52.001 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1053000000000002e-05 2024-09-16T04:42:52.002 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.002 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.002 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.002 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.002 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-09-16T04:42:52.002 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 3 2024-09-16T04:42:52.002 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:52.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:42:52.024 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:52.019+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:52.124 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:52.119+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:52.130 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:52.131 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:52.131 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:52.131 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:52.131 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:52.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:42:52.226 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:52.223+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:52.268 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:52.268 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:52.268 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:148 3.4s0 3:2b4d2b94:::smithi16530038-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 66.065325044999994, 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.49811553600000003, 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.269 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 148 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198240+0000", 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3600000000000003e-07 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198243+0000", 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.281e-06 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198250+0000", 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7870000000000004e-06 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.271 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298324+0000", 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100073891 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298376+0000", 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2790000000000001e-05 2024-09-16T04:42:52.272 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298437+0000", 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0550999999999997e-05 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298723+0000", 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000285741 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.273 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.299721+0000", 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00099841899999999996 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696249+0000", 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39652776099999998 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696354+0000", 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105679 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.274 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:149 3.4s0 3:2b4d2b94:::smithi16530038-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 65.965212566999995, 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.59985384399999997, 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.275 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 149 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.276 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298354+0000", 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298362+0000", 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3319999999999992e-06 2024-09-16T04:42:52.277 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298377+0000", 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5387999999999999e-05 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298388+0000", 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0957e-05 2024-09-16T04:42:52.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398555+0000", 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016630999999999 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398706+0000", 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015114699999999999 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696493+0000", 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29778684 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696643+0000", 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015093699999999999 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.898205+0000", 2024-09-16T04:42:52.280 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20156174700000001 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:160 3.4s0 3:2d731f27:::smithi16530038-30:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 65.809582500999994, 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44291274600000002, 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 160 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.282 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453983+0000", 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4729999999999999e-06 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453989+0000", 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1299999999999998e-06 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454005+0000", 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5880999999999999e-05 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454014+0000", 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1079999999999999e-06 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454090+0000", 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6062999999999993e-05 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454166+0000", 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5772999999999994e-05 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454458+0000", 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029229599999999999 2024-09-16T04:42:52.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.455106+0000", 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00064800899999999995 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896766+0000", 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44165951199999998 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.286 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896894+0000", 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000128501 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.14s0 42/41 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.430863785, 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60655789999999998, 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.287 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832701+0000", 2024-09-16T04:42:52.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6400000000000001e-07 2024-09-16T04:42:52.289 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.289 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.289 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.289 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832733+0000", 2024-09-16T04:42:52.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2382000000000003e-05 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832757+0000", 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3691999999999999e-05 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.932887+0000", 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100130238 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.438947+0000", 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50605998399999996 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439258+0000", 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031074 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.291 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=56, reqid=osd.11.0:56, at_version=44'19, trim_to=0'0, roll_forward_to=39'17))", 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 58.144669337000003, 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39966409400000003, 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.292 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.293 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118895+0000", 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6400000000000003e-07 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118922+0000", 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6635000000000002e-05 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118941+0000", 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9707999999999998e-05 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.118952+0000", 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.039e-05 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.513442+0000", 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39449013100000002 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.513459+0000", 2024-09-16T04:42:52.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6748e-05 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.518526+0000", 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0050673970000000004 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:54.518559+0000", 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2620999999999998e-05 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.296 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.14.22:1 3.4s0 3:208a5e75:::smithi16530038-37:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 53.602639197999999, 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40385833399999999, 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.297 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660925+0000", 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660929+0000", 2024-09-16T04:42:52.298 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7859999999999998e-06 2024-09-16T04:42:52.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.299 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.299 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660942+0000", 2024-09-16T04:42:52.299 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3526999999999999e-05 2024-09-16T04:42:52.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.660996+0000", 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3860999999999998e-05 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.761100+0000", 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100103686 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.857146+0000", 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.096046099999999995 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:58.857321+0000", 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000175465 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.064783+0000", 2024-09-16T04:42:52.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.207461909 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:368 3.14s0 3:28a14731:::smithi16530038-3:head [setxattr _header (55) in=7b] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 53.078523091999998, 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50583271100000005, 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.302 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 368 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.303 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.304 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185043+0000", 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.463e-06 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185054+0000", 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0356e-05 2024-09-16T04:42:52.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185080+0000", 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6519000000000001e-05 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185097+0000", 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.717e-05 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.188751+0000", 2024-09-16T04:42:52.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036535249999999999 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.189019+0000", 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026812599999999999 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.190416+0000", 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013973830000000001 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.389174+0000", 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19875738700000001 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690722+0000", 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30154792000000002 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690874+0000", 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000151862 2024-09-16T04:42:52.308 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:369 3.14s0 3:28a14731:::smithi16530038-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 53.078441374999997, 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.81195188399999996, 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.309 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 369 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.310 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185124+0000", 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.33e-06 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185130+0000", 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9070000000000004e-06 2024-09-16T04:42:52.311 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185154+0000", 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4059000000000001e-05 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185173+0000", 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8936e-05 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.312 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388708+0000", 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.203534774 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388994+0000", 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028647200000000002 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691636+0000", 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.302641681 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691904+0000", 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026824200000000002 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.997074+0000", 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30517048299999999 2024-09-16T04:42:52.314 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:618 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [append 0~0] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.894264815, 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.54505251200000004, 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.315 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 618 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:42:52.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369302+0000", 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6639999999999998e-06 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369315+0000", 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3416e-05 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.317 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369345+0000", 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9961e-05 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369359+0000", 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4107000000000001e-05 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.604180+0000", 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.23482078300000001 2024-09-16T04:42:52.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.604381+0000", 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020054599999999999 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.605620+0000", 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012394699999999999 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.608601+0000", 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029811059999999999 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914102+0000", 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.305500773 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914352+0000", 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024968599999999999 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:619 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [rollback 11] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.894177306, 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.54536771900000003, 2024-09-16T04:42:52.321 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 619 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.68e-07 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369390+0000", 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.948e-06 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369399+0000", 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3499999999999997e-06 2024-09-16T04:42:52.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369405+0000", 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.455e-06 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.606687+0000", 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.237281837 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.606930+0000", 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024356099999999999 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.607700+0000", 2024-09-16T04:42:52.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000769507 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.810765+0000", 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20306539400000001 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914602+0000", 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103836502 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914754+0000", 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000152597 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:620 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [append 0~0] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.894151548, 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.81742312399999995, 2024-09-16T04:42:52.328 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 620 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:42:52.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:42:52.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369413+0000", 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9099999999999999e-07 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369426+0000", 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.365e-05 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369435+0000", 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4700000000000002e-06 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.469519+0000", 2024-09-16T04:42:52.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100084346 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.912089+0000", 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44256948200000001 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.912270+0000", 2024-09-16T04:42:52.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018106700000000001 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.913255+0000", 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098478400000000005 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914789+0000", 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015345210000000001 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.186759+0000", 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27197024199999997 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.186836+0000", 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6171e-05 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:706 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.861944+0000", 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.401619615000001, 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40975806300000001, 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.336 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 706 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861944+0000", 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861944+0000", 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861946+0000", 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9099999999999999e-06 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861955+0000", 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1330000000000005e-06 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.861979+0000", 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3779000000000001e-05 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862007+0000", 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7628999999999999e-05 2024-09-16T04:42:52.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862157+0000", 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015022200000000001 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862408+0000", 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025079200000000001 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.863488+0000", 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001080592 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.070476+0000", 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.206987744 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271563+0000", 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20108668199999999 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271702+0000", 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013957999999999999 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:707 3.1es0 3:78f34faf:::smithi16530038-13:head [rollback 11] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.401531081999998, 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51046276300000004, 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.342 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 707 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862034+0000", 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:42:52.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862038+0000", 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9160000000000003e-06 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862049+0000", 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1786e-05 2024-09-16T04:42:52.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862057+0000", 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.966992+0000", 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10493451600000001 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.967212+0000", 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021991 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.968498+0000", 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012864350000000001 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271112+0000", 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30261409299999997 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372370+0000", 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10125774899999999 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372496+0000", 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000125501 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:708 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.401449988, 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.74368462499999999, 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 708 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.349 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:42:52.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7800000000000002e-07 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862116+0000", 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.08e-06 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862122+0000", 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7930000000000003e-06 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862126+0000", 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2470000000000002e-06 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373611+0000", 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51148433299999996 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373943+0000", 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00033231500000000002 2024-09-16T04:42:52.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.375178+0000", 2024-09-16T04:42:52.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001235367 2024-09-16T04:42:52.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.376541+0000", 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013628410000000001 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.605718+0000", 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22917667899999999 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.605799+0000", 2024-09-16T04:42:52.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0592000000000004e-05 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=71, last_complete=79'55, committed=1, applied=1))", 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.866045+0000", 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.397518992999998, 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40485729799999998, 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.358 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866045+0000", 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866045+0000", 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866046+0000", 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.37e-07 2024-09-16T04:42:52.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866052+0000", 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3520000000000003e-06 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866061+0000", 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5300000000000002e-06 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866067+0000", 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5740000000000003e-06 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.270839+0000", 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40477243800000001 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.270902+0000", 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2767000000000004e-05 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.361 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=72, last_complete=79'57, committed=1, applied=1))", 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.866722+0000", 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.396841544000001, 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40500170400000002, 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866722+0000", 2024-09-16T04:42:52.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866722+0000", 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866723+0000", 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7400000000000002e-07 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866728+0000", 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4229999999999998e-06 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866738+0000", 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0876000000000001e-05 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.866745+0000", 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6669999999999999e-06 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271420+0000", 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40467444499999999 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271724+0000", 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000304519 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=72, last_complete=79'57, committed=1, applied=1))", 2024-09-16T04:42:52.365 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.396522209, 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40407422999999998, 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867042+0000", 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867046+0000", 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8739999999999997e-06 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867060+0000", 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4056999999999999e-05 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.867067+0000", 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7560000000000002e-06 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271073+0000", 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40400594099999998 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271116+0000", 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2970999999999999e-05 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=73, last_complete=79'58, committed=1, applied=1))", 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.970076+0000", 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.293488177, 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40209012, 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.369 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970076+0000", 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970076+0000", 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970078+0000", 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.827e-06 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970088+0000", 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0769e-05 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970114+0000", 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5839000000000001e-05 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970127+0000", 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3118999999999999e-05 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372126+0000", 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40199895699999999 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372166+0000", 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9609e-05 2024-09-16T04:42:52.372 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 79/69 ECSubWriteReply(tid=73, last_complete=79'58, committed=1, applied=1))", 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.970504+0000", 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.293059964000001, 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40299647599999999, 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970504+0000", 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970504+0000", 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970505+0000", 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.173e-06 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970516+0000", 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1270000000000001e-05 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.970538+0000", 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1622000000000001e-05 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.070638+0000", 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100099904 2024-09-16T04:42:52.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372240+0000", 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30160231700000001 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373500+0000", 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00126019 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 92/67 ECSubWrite(tid=156, reqid=osd.2.0:156, at_version=92'88, trim_to=0'0, roll_forward_to=90'84))", 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:42.515434+0000", 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.7481301360000003, 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.49799755800000001, 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515434+0000", 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515434+0000", 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515435+0000", 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1629999999999999e-06 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515466+0000", 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0927999999999999e-05 2024-09-16T04:42:52.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515487+0000", 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0792e-05 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.615560+0000", 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10007328 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.011802+0000", 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.396242135 2024-09-16T04:42:52.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.011814+0000", 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.216e-05 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.013372+0000", 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015577449999999999 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.013431+0000", 2024-09-16T04:42:52.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9354999999999997e-05 2024-09-16T04:42:52.381 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.381 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.381 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.381 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.381 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.381 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:52.382 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:42:52.383 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:52.375+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:52.484 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:52.479+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:52.513 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:52.513 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:52.513 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:42:52.514 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:52.528 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:42:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:marked out osd.15. 2024-09-16T04:42:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:370: done (5 left) 2024-09-16T04:42:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:378: done (4 left) 2024-09-16T04:42:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:381: done (3 left) 2024-09-16T04:42:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 9 current snap is 34 2024-09-16T04:42:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:382: seq_num 140 ranges {2424832=802816,3227648=425984,3653632=720896,4374528=409600} 2024-09-16T04:42:52.589 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:52.579+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:52.607 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.15 2024-09-16T04:42:52.608 INFO:tasks.thrashosds.thrasher:remote for osd 15 is ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:42:52.608 DEBUG:teuthology.orchestra.run.smithi165:> type ceph-objectstore-tool 2024-09-16T04:42:52.613 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:42:52.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-15 --journal-path /var/lib/ceph/osd/ceph-15/journal --op list-pgs 2024-09-16T04:42:52.638 INFO:tasks.rados.rados.0.smithi165.stdout:382: writing smithi16530038-9 from 2424832 to 3227648 tid 1 2024-09-16T04:42:52.644 INFO:tasks.rados.rados.0.smithi165.stdout:382: writing smithi16530038-9 from 3227648 to 3653632 tid 2 2024-09-16T04:42:52.652 INFO:tasks.rados.rados.0.smithi165.stdout:382: writing smithi16530038-9 from 3653632 to 4374528 tid 3 2024-09-16T04:42:52.656 INFO:tasks.rados.rados.0.smithi165.stdout:382: writing smithi16530038-9 from 4374528 to 4784128 tid 4 2024-09-16T04:42:52.656 INFO:tasks.rados.rados.0.smithi165.stdout:383: setattr oid 4 current snap is 34 2024-09-16T04:42:52.658 INFO:tasks.rados.rados.0.smithi165.stdout:384: snap_create 2024-09-16T04:42:52.661 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:52.661 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:42:52.661 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:42:52.661 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:42:52.661 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:52.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:42:52.685 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:52.679+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:52.786 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:52.783+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=29, reqid=client.4728.0:365, at_version=49'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 53.531681790999997, 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15627164299999999, 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.806 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271408+0000", 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271409+0000", 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271765+0000", 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00035584100000000002 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271795+0000", 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9595999999999999e-05 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.271820+0000", 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5301999999999999e-05 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425575+0000", 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.15375470799999999 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.425593+0000", 2024-09-16T04:42:52.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8249000000000001e-05 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427644+0000", 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020504630000000002 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.427680+0000", 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6613999999999999e-05 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 53/32 ECSubWrite(tid=86, reqid=client.4728.0:424, at_version=53'27, trim_to=0'0, roll_forward_to=49'25))", 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 49.463574110000003, 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10390139399999999, 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339516+0000", 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339517+0000", 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.167e-06 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339944+0000", 2024-09-16T04:42:52.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00042670199999999999 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339971+0000", 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6916000000000001e-05 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.339982+0000", 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1445e-05 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.440244+0000", 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100261356 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.440271+0000", 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7209e-05 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.443349+0000", 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030780949999999999 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.443418+0000", 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8503999999999993e-05 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=118, reqid=client.4728.0:458, at_version=58'27, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 44.397675778999997, 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20346340800000001, 2024-09-16T04:42:52.815 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405414+0000", 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405416+0000", 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.221e-06 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405551+0000", 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013492800000000001 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405567+0000", 2024-09-16T04:42:52.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6552999999999999e-05 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505670+0000", 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10010258399999999 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505840+0000", 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017047199999999999 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505892+0000", 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1406000000000002e-05 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608838+0000", 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10294645299999999 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608878+0000", 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9790999999999999e-05 2024-09-16T04:42:52.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 58/32 ECSubWrite(tid=120, reqid=client.4728.0:460, at_version=58'29, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 44.297145026000003, 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10344765, 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.820 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505945+0000", 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505946+0000", 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.505979+0000", 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3062000000000003e-05 2024-09-16T04:42:52.822 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.506027+0000", 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7333000000000002e-05 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.506050+0000", 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2918999999999999e-05 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608654+0000", 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102604337 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.608663+0000", 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.986e-06 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.609362+0000", 2024-09-16T04:42:52.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00069878900000000001 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.609393+0000", 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1241000000000001e-05 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/32 ECSubWrite(tid=132, reqid=osd.4.0:132, at_version=60'35, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:52.825 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.340221638000003, 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20257109500000001, 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462869+0000", 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3300000000000003e-07 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.827 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462925+0000", 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6362e-05 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462958+0000", 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2876000000000002e-05 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.828 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.462991+0000", 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2836000000000002e-05 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463182+0000", 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000190736 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463210+0000", 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7925000000000001e-05 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665388+0000", 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202177734 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665440+0000", 2024-09-16T04:42:52.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2193e-05 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.339263399000004, 2024-09-16T04:42:52.831 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20164931799999999, 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.832 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463827+0000", 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463828+0000", 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.161e-06 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463862+0000", 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3846000000000001e-05 2024-09-16T04:42:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463881+0000", 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9596e-05 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.463895+0000", 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3448999999999999e-05 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.464025+0000", 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000129677 2024-09-16T04:42:52.834 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.464038+0000", 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3654e-05 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665451+0000", 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201412806 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:10.665476+0000", 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5128999999999999e-05 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 65/32 ECSubWrite(tid=59, reqid=osd.2.0:59, at_version=65'40, trim_to=0'0, roll_forward_to=64'36))", 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 37.351110345999999, 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202052495, 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.836 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451980+0000", 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.837 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.451982+0000", 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.227e-06 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452027+0000", 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5142000000000002e-05 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452061+0000", 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3529999999999999e-05 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.838 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.452079+0000", 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7941999999999999e-05 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.652452+0000", 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200372941 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.652468+0000", 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6517000000000001e-05 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.653953+0000", 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001484857 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.654032+0000", 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9338999999999994e-05 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.840 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=75, reqid=client.4728.0:577, at_version=69'50, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.474072606999997, 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20012123100000001, 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329018+0000", 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329019+0000", 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.654e-06 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329563+0000", 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054386599999999995 2024-09-16T04:42:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329591+0000", 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7759999999999998e-05 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.329619+0000", 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7968999999999999e-05 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427652+0000", 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098032790999999994 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.427662+0000", 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0736e-05 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529104+0000", 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10144188 2024-09-16T04:42:52.844 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529139+0000", 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4575e-05 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=76, reqid=client.4728.0:578, at_version=69'51, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.464965843999998, 2024-09-16T04:42:52.845 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19168868999999999, 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338124+0000", 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.846 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338125+0000", 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9099999999999991e-07 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338313+0000", 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018766200000000001 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338329+0000", 2024-09-16T04:42:52.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5807e-05 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.338339+0000", 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0451e-05 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.528499+0000", 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19015979599999999 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.848 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.528508+0000", 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0469999999999999e-06 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529785+0000", 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001276758 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529813+0000", 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8178e-05 2024-09-16T04:42:52.849 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 69/67 ECSubWrite(tid=77, reqid=client.4728.0:579, at_version=69'52, trim_to=0'0, roll_forward_to=68'49))", 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.461133273000002, 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18830155900000001, 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.850 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341957+0000", 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.341958+0000", 2024-09-16T04:42:52.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0400000000000005e-07 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342250+0000", 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029209100000000002 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342265+0000", 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4808e-05 2024-09-16T04:42:52.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.342273+0000", 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3760000000000008e-06 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529207+0000", 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18693383999999999 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.529216+0000", 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9109999999999999e-06 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.530224+0000", 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00100815 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.530259+0000", 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4579000000000002e-05 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.855 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 70/67 ECSubWrite(tid=152, reqid=osd.4.0:152, at_version=70'50, trim_to=0'0, roll_forward_to=69'48))", 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.482453577999998, 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.14250185500000001, 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.856 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320637+0000", 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5300000000000004e-07 2024-09-16T04:42:52.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320696+0000", 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8539000000000003e-05 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320732+0000", 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6368999999999997e-05 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320759+0000", 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6466999999999999e-05 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320930+0000", 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017177699999999999 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.320966+0000", 2024-09-16T04:42:52.859 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5382000000000001e-05 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463102+0000", 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.14213593499999999 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463139+0000", 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6733000000000002e-05 2024-09-16T04:42:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 70/67 ECSubWrite(tid=153, reqid=osd.4.0:153, at_version=70'52, trim_to=0'0, roll_forward_to=69'48))", 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:20.321873+0000", 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.481217676999997, 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.14129620000000001, 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.861 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321873+0000", 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321873+0000", 2024-09-16T04:42:52.862 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321875+0000", 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.03e-06 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321927+0000", 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2876e-05 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.863 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321958+0000", 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0960000000000002e-05 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.321981+0000", 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2742e-05 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.422274+0000", 2024-09-16T04:42:52.864 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10029260199999999 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.422289+0000", 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5319999999999999e-05 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463146+0000", 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.040857346000000003 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:20.463169+0000", 2024-09-16T04:42:52.865 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2323999999999998e-05 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs3 71/67 ECSubWrite(tid=156, reqid=osd.4.0:156, at_version=71'54, trim_to=0'0, roll_forward_to=70'52))", 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 31.644009532999998, 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201114086, 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.866 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.867 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159081+0000", 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0400000000000002e-07 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159105+0000", 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3816999999999999e-05 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.868 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.159199+0000", 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3851999999999995e-05 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.259300+0000", 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100100827 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.359431+0000", 2024-09-16T04:42:52.869 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131028 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.359440+0000", 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3300000000000005e-06 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.360143+0000", 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00070308700000000003 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.870 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:21.360195+0000", 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1841000000000001e-05 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 74/67 ECSubWrite(tid=97, reqid=client.4728.0:613, at_version=74'68, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.370209+0000", 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 28.432881169000002, 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20209957100000001, 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.871 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370209+0000", 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370209+0000", 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.872 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370210+0000", 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2300000000000002e-07 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370720+0000", 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000510762 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.370739+0000", 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8434e-05 2024-09-16T04:42:52.873 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.470817+0000", 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10007798699999999 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.470878+0000", 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.109e-05 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.470888+0000", 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0392e-05 2024-09-16T04:42:52.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.572280+0000", 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10139131899999999 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.572309+0000", 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9164000000000001e-05 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.875 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.bs3,RELEASE,e:75,from:4(0),reservation_nonce:0)", 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:25.281000+0000", 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.522089899000001, 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.198626372, 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281000+0000", 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.876 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281000+0000", 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281001+0000", 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3499999999999995e-07 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281005+0000", 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.245e-06 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281019+0000", 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3633e-05 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.281027+0000", 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7309999999999993e-06 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.479556+0000", 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19852925199999999 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.479595+0000", 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9357000000000003e-05 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.479627+0000", 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1418999999999999e-05 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.879 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.1bs4,QREQUEST,e:75,from:10(0),reservation_nonce:5)", 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:25.349066+0000", 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 27.454024726, 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13250140399999999, 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.880 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349066+0000", 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349066+0000", 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349068+0000", 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7130000000000001e-06 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349082+0000", 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3593e-05 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349120+0000", 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7858999999999998e-05 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.349137+0000", 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7303e-05 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.481493+0000", 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13235649399999999 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.481518+0000", 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4783999999999999e-05 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.481567+0000", 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8658000000000001e-05 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.883 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 90/67 ECSubWrite(tid=230, reqid=osd.11.0:230, at_version=90'81, trim_to=0'0, roll_forward_to=82'79))", 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:41.007422+0000", 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.795667927, 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20141141300000001, 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.884 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.007422+0000", 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.007422+0000", 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.007424+0000", 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.007e-06 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.885 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.007468+0000", 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3438e-05 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.007500+0000", 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2252000000000002e-05 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.107612+0000", 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011199799999999 2024-09-16T04:42:52.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.107984+0000", 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00037174799999999998 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.108019+0000", 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5111999999999999e-05 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.208748+0000", 2024-09-16T04:42:52.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100728687 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:41.208834+0000", 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6170999999999999e-05 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 96/67 ECSubWrite(tid=248, reqid=client.4728.0:890, at_version=96'89, trim_to=0'0, roll_forward_to=92'87))", 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:46.547152+0000", 2024-09-16T04:42:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.2559384040000001, 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.106319993, 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547152+0000", 2024-09-16T04:42:52.889 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547152+0000", 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547152+0000", 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7700000000000004e-07 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.890 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.891 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547725+0000", 2024-09-16T04:42:52.891 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00057234199999999999 2024-09-16T04:42:52.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.891 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.891 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547763+0000", 2024-09-16T04:42:52.891 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8325999999999999e-05 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547790+0000", 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7056e-05 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547958+0000", 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000167625 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.547985+0000", 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7639000000000001e-05 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.653404+0000", 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10541889 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.893 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:46.653472+0000", 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7538000000000003e-05 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDPGPush(3.1es6 101/97 [PushOp(3:78f34faf:::smithi16530038-13:head, version: 95'85, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:78f34faf:::smithi16530038-13:head@95'85, size: 3915776, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:51.821085+0000", 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.98200484899999996, 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20188350399999999, 2024-09-16T04:42:52.894 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.821085+0000", 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.821085+0000", 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.821086+0000", 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.85e-07 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.896 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.821806+0000", 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072008399999999996 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.821833+0000", 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6662999999999999e-05 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.897 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.921954+0000", 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100120742 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.022245+0000", 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10029110199999999 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.022969+0000", 2024-09-16T04:42:52.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072412799999999999 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDPGPush(3.1es6 101/97 [PushOp(3:7ee6b8f9:::smithi16530038-8:1c, version: 95'86, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:7ee6b8f9:::smithi16530038-8:1c@95'86, size: 1376256, copy_subset: [], clone_subset: {}, snapset: 20=[]:{1c: [1b, 1a, 19, 18]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:51.942956+0000", 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.860134328, 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.18121705799999999, 2024-09-16T04:42:52.899 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.942956+0000", 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.942956+0000", 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.942957+0000", 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2410000000000001e-06 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.943579+0000", 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00062196300000000001 2024-09-16T04:42:52.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.943613+0000", 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3895e-05 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.943636+0000", 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2476999999999998e-05 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.123165+0000", 2024-09-16T04:42:52.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17952991900000001 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.124173+0000", 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010075629999999999 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:42:52.903 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:42:52.904 INFO:tasks.rados.rados.0.smithi165.stdout:382: finishing write tid 1 to smithi16530038-9 2024-09-16T04:42:52.905 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:42:52.906 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:52.899+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:52.912 INFO:tasks.rados.rados.0.smithi165.stdout:368: finishing write tid 1 to smithi16530038-11 2024-09-16T04:42:52.915 INFO:tasks.rados.rados.0.smithi165.stdout:368: finishing write tid 2 to smithi16530038-11 2024-09-16T04:42:52.954 INFO:tasks.rados.rados.0.smithi165.stdout:382: finishing write tid 2 to smithi16530038-9 2024-09-16T04:42:53.005 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:53.003+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:53.026 INFO:tasks.rados.rados.0.smithi165.stdout:368: finishing write tid 3 to smithi16530038-11 2024-09-16T04:42:53.028 INFO:tasks.rados.rados.0.smithi165.stdout:368: finishing write tid 4 to smithi16530038-11 2024-09-16T04:42:53.036 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:53.036 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:53.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.036 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1es3 102/97 ECSubRead(tid=195, to_read={3:78f34faf:::smithi16530038-13:head=786432,262144,0}, subchunks={3:78f34faf:::smithi16530038-13:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:53.036 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:52.956621+0000", 2024-09-16T04:42:53.036 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.079338786999999994, 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.079371405000000006, 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:52.956621+0000", 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:52.956621+0000", 2024-09-16T04:42:53.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:52.956622+0000", 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1149999999999999e-06 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:52.956636+0000", 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.411e-05 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:52.956653+0000", 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7337e-05 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:52.956663+0000", 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6369999999999994e-06 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 1 2024-09-16T04:42:53.039 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:53.040 INFO:tasks.rados.rados.0.smithi165.stdout:368: finishing write tid 5 to smithi16530038-11 2024-09-16T04:42:53.051 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:42:53.106 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:53.102+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:53.177 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:53.177 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:53.177 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:53.177 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:53.177 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:53.187 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:42:53.207 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:53.202+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:53.260 INFO:tasks.rados.rados.0.smithi165.stdout:382: finishing write tid 3 to smithi16530038-9 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=10, reqid=client.4728.0:122, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 67.622695411999999, 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201524707, 2024-09-16T04:42:53.309 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684051+0000", 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.277e-06 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684081+0000", 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0414999999999999e-05 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684109+0000", 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7455000000000001e-05 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684125+0000", 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5707000000000001e-05 2024-09-16T04:42:53.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884504+0000", 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200379487 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884519+0000", 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4953000000000001e-05 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885504+0000", 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098537600000000005 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885574+0000", 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0036999999999994e-05 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 67.332373813000004, 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20260964000000001, 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:53.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974373+0000", 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.683e-06 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974841+0000", 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468255 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974891+0000", 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0198000000000003e-05 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.074965+0000", 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100073226 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175170+0000", 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100205119 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175184+0000", 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4626e-05 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176898+0000", 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017136390000000001 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176981+0000", 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2893999999999996e-05 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:160, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 66.853082203, 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20191688999999999, 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453664+0000", 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453695+0000", 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0998999999999999e-05 2024-09-16T04:42:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453714+0000", 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9745000000000002e-05 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.553888+0000", 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10017351100000001 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654137+0000", 2024-09-16T04:42:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100248949 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654151+0000", 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4279e-05 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655526+0000", 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013750520000000001 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655580+0000", 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3634999999999997e-05 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 48/32 ECSubWrite(tid=44, reqid=client.4728.0:362, at_version=48'17, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 54.950058902999999, 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205456467, 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356687+0000", 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356985+0000", 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029765100000000001 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357014+0000", 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9853e-05 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357034+0000", 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9592000000000001e-05 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557308+0000", 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20027399500000001 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557322+0000", 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4346e-05 2024-09-16T04:42:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562072+0000", 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047490960000000004 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562143+0000", 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1191000000000003e-05 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s6 57/53 ECSubWrite(tid=9, reqid=osd.10.0:9, at_version=57'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 45.077627995999997, 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209615832, 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229119+0000", 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229208+0000", 2024-09-16T04:42:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9190999999999994e-05 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229249+0000", 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0842e-05 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329348+0000", 2024-09-16T04:42:53.331 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10009905600000001 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426747+0000", 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.09739892 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426767+0000", 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0173000000000002e-05 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438659+0000", 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011892171 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438733+0000", 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3515000000000003e-05 2024-09-16T04:42:53.333 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s3 63/55 ECSubWrite(tid=22, reqid=osd.5.0:22, at_version=63'14, trim_to=0'0, roll_forward_to=62'12))", 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 39.728348855, 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20251555199999999, 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.334 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9999999999999999e-07 2024-09-16T04:42:53.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.336 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.336 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578420+0000", 2024-09-16T04:42:53.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3271000000000001e-05 2024-09-16T04:42:53.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578449+0000", 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9488999999999998e-05 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.678595+0000", 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100145814 2024-09-16T04:42:53.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778824+0000", 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100228703 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778840+0000", 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6123000000000001e-05 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.338 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780838+0000", 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019980129999999999 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780912+0000", 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3938999999999998e-05 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.339 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es2 69/67 ECSubWrite(tid=80, reqid=client.4728.0:581, at_version=0'0, trim_to=0'0, roll_forward_to=69'54))", 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 33.580174141999997, 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20154364599999999, 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.340 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726573+0000", 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1399999999999998e-06 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726628+0000", 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4376000000000003e-05 2024-09-16T04:42:53.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726668+0000", 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0466000000000002e-05 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.826832+0000", 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100163697 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.342 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827016+0000", 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018461700000000001 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827048+0000", 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2186e-05 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928006+0000", 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10095765900000001 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928115+0000", 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010850500000000001 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1es3 74/69 ECSubWrite(tid=62, reqid=client.4728.0:620, at_version=74'53, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 28.322894980000001, 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20195012000000001, 2024-09-16T04:42:53.344 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983851+0000", 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.173e-06 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983905+0000", 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4027000000000003e-05 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.346 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983927+0000", 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2005999999999999e-05 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084064+0000", 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10013664899999999 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084267+0000", 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020252399999999999 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.347 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084288+0000", 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1943999999999999e-05 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.185701+0000", 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101412144 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.185800+0000", 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9653000000000003e-05 2024-09-16T04:42:53.348 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:697 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~704512] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.815839462, 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.26883937299999999, 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.349 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 697 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490907+0000", 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8050000000000001e-06 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491858+0000", 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095084100000000001 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.351 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491869+0000", 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1184e-05 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491878+0000", 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3480000000000004e-06 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491963+0000", 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5368e-05 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491974+0000", 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1161e-05 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492735+0000", 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00076025099999999998 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492939+0000", 2024-09-16T04:42:53.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020470800000000001 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.494899+0000", 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019597260000000002 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.496840+0000", 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019404660000000001 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:53.354 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759668+0000", 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.262828544 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759745+0000", 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6971000000000006e-05 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:698 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~655360] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.814844502, 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.36985094099999999, 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 698 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.356 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491902+0000", 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9100000000000002e-07 2024-09-16T04:42:53.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492632+0000", 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00073060200000000001 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492643+0000", 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0910999999999999e-05 2024-09-16T04:42:53.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592713+0000", 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100069806 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593257+0000", 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00054396700000000002 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593384+0000", 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012739399999999999 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.594523+0000", 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011388450000000001 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759798+0000", 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.16527468300000001 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.861671+0000", 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101873753 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.861752+0000", 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0088999999999998e-05 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.361 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:699 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~507904] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.714001845999999, 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.37090961900000002, 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 699 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592745+0000", 2024-09-16T04:42:53.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7e-06 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593160+0000", 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041519899999999999 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593172+0000", 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2248e-05 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593182+0000", 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4210000000000001e-06 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.595327+0000", 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0021456460000000002 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.595409+0000", 2024-09-16T04:42:53.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1316000000000005e-05 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.596547+0000", 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011381710000000001 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759799+0000", 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.163252014 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963555+0000", 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203755827 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963653+0000", 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8077000000000003e-05 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:700 3.1fs0 3:fb308b36:::smithi16530038-23:head [setxattr _header (57) in=7b] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:53.367 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.713543485999999, 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.37068284299999998, 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 700 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.368 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593207+0000", 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7740000000000004e-06 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593217+0000", 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9329999999999996e-06 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593225+0000", 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4749999999999996e-06 2024-09-16T04:42:53.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.597569+0000", 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0043444690000000001 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.597710+0000", 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141125 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.598396+0000", 2024-09-16T04:42:53.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000685767 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759801+0000", 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.16140457599999999 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963785+0000", 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20398479 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963884+0000", 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9166999999999994e-05 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:701 3.1fs0 3:fb308b36:::smithi16530038-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.713499437999999, 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.67642940299999998, 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.373 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 701 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.51e-07 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593250+0000", 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0099999999999997e-06 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593260+0000", 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6360000000000006e-06 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593267+0000", 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2869999999999999e-06 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.600494+0000", 2024-09-16T04:42:53.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072266040000000002 2024-09-16T04:42:53.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.377 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:53.377 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.600669+0000", 2024-09-16T04:42:53.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000174799 2024-09-16T04:42:53.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.964260+0000", 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.36359097400000001 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.964360+0000", 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000100375 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.269675+0000", 2024-09-16T04:42:53.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.305315167 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s3 79/53 ECSubWrite(tid=126, reqid=client.4728.0:704, at_version=79'66, trim_to=0'0, roll_forward_to=72'63))", 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.54196584, 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20206831, 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764781+0000", 2024-09-16T04:42:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.796e-06 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764820+0000", 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8662999999999999e-05 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764850+0000", 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0349999999999999e-05 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.864962+0000", 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100111618 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965141+0000", 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100179383 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965156+0000", 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4997000000000001e-05 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.966786+0000", 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016294300000000001 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.966848+0000", 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2073e-05 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1fs0 79/69 ECSubWriteReply(tid=4, last_complete=79'56, committed=1, applied=1))", 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:42:53.386 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.341109071999998, 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202173098, 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:42:53.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965637+0000", 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9400000000000005e-07 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965640+0000", 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4130000000000002e-06 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965650+0000", 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0207000000000001e-05 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965659+0000", 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6759999999999996e-06 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167760+0000", 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202100745 2024-09-16T04:42:53.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167809+0000", 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9462999999999997e-05 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1fs0 79/69 ECSubWriteReply(tid=4, last_complete=79'56, committed=1, applied=1))", 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.340970293000002, 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202105168, 2024-09-16T04:42:53.390 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965777+0000", 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6700000000000001e-06 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965789+0000", 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2173e-05 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965816+0000", 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7270000000000001e-05 2024-09-16T04:42:53.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965837+0000", 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.107e-05 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167854+0000", 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20201680599999999 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167880+0000", 2024-09-16T04:42:53.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6179000000000001e-05 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1fs0 79/69 ECSubRead(tid=5, to_read={3:fb308b36:::smithi16530038-23:head=0,4096,0}, subchunks={3:fb308b36:::smithi16530038-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.339545192999999, 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20164594899999999, 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.394 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:53.395 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967275+0000", 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4894999999999994e-05 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.067391+0000", 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100115909 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.168332+0000", 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100941066 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.168846+0000", 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051407899999999997 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.397 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 80/67 ECSubWrite(tid=131, reqid=osd.3.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=80'51))", 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 22.60958157, 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30121982800000002, 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.398 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697165+0000", 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.623e-06 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697185+0000", 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0103e-05 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697215+0000", 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9733000000000001e-05 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697231+0000", 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5699999999999999e-05 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.897603+0000", 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20037271700000001 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.897620+0000", 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6983999999999999e-05 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.998295+0000", 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100674176 2024-09-16T04:42:53.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.998383+0000", 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8792e-05 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPush(3.19s6 95/58 [PushOp(3:9de467d5:::smithi16530038-27:7, version: 50'27, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:9de467d5:::smithi16530038-27:7@50'27, size: 1294336, copy_subset: [], clone_subset: {}, snapset: 11=[]:{4: [3], 7: [7], 11: [11, d, b]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:46.633350+0000", 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.6733948309999995, 2024-09-16T04:42:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205371676, 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633350+0000", 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633350+0000", 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633351+0000", 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3399999999999999e-07 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633649+0000", 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029821699999999998 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633711+0000", 2024-09-16T04:42:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2002999999999996e-05 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.733901+0000", 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100189615 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.837764+0000", 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10386297899999999 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.838722+0000", 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095832800000000004 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1es2 100/97 ECSubWrite(tid=188, reqid=client.4728.0:926, at_version=0'0, trim_to=0'0, roll_forward_to=99'92))", 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:51.288513+0000", 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.0182321750000001, 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20181585499999999, 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288513+0000", 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288513+0000", 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288515+0000", 2024-09-16T04:42:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7e-06 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288545+0000", 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9865999999999998e-05 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288575+0000", 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0329999999999999e-05 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.388711+0000", 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10013585799999999 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.489095+0000", 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10038437 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.489112+0000", 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7022999999999999e-05 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.490243+0000", 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001130806 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.490329+0000", 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5902000000000005e-05 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.410 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:53.411 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:53.303+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:53.412 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:42:53.413 INFO:tasks.rados.rados.0.smithi165.stdout:368: finishing write tid 6 to smithi16530038-11 2024-09-16T04:42:53.413 INFO:tasks.rados.rados.0.smithi165.stdout:368: oid 11 updating version 0 to 160 2024-09-16T04:42:53.413 INFO:tasks.rados.rados.0.smithi165.stdout:368: oid 11 updating version 160 to 162 2024-09-16T04:42:53.413 INFO:tasks.rados.rados.0.smithi165.stdout:368: oid 11 updating version 162 to 163 2024-09-16T04:42:53.413 INFO:tasks.rados.rados.0.smithi165.stdout:368: oid 11 version 163 is already newer than 161 2024-09-16T04:42:53.413 INFO:tasks.rados.rados.0.smithi165.stdout:368: oid 11 version 163 is already newer than 159 2024-09-16T04:42:53.414 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 163 (ObjNum 137 snap 31 seq_num 137) dirty exists 2024-09-16T04:42:53.414 INFO:tasks.rados.rados.0.smithi165.stdout:368: left oid 11 (ObjNum 137 snap 31 seq_num 137) 2024-09-16T04:42:53.414 INFO:tasks.rados.rados.0.smithi165.stdout:382: finishing write tid 4 to smithi16530038-9 2024-09-16T04:42:53.414 INFO:tasks.rados.rados.0.smithi165.stdout:382: finishing write tid 5 to smithi16530038-9 2024-09-16T04:42:53.414 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:53.411+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:53.514 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:53.511+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:53.587 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:53.587 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:53.587 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.587 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1es2 102/97 ECSubRead(tid=196, to_read={3:78f34faf:::smithi16530038-13:1d=0,262144,0}, subchunks={3:78f34faf:::smithi16530038-13:1d=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:53.587 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:53.574826+0000", 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.011709694, 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011760883, 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:53.574826+0000", 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:53.574826+0000", 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:53.574828+0000", 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.962e-06 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:53.574840+0000", 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2191999999999999e-05 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:53.574865+0000", 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5134e-05 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:53.574902+0000", 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6763e-05 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.590 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.591 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.591 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.591 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-09-16T04:42:53.591 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 1 2024-09-16T04:42:53.591 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:53.591 INFO:tasks.rados.rados.0.smithi165.stdout:368: done (5 left) 2024-09-16T04:42:53.592 INFO:tasks.rados.rados.0.smithi165.stdout:384: done (4 left) 2024-09-16T04:42:53.592 INFO:tasks.rados.rados.0.smithi165.stdout:385: copy_from oid 26 from oid 5 current snap is 35 2024-09-16T04:42:53.592 INFO:tasks.rados.rados.0.smithi165.stdout:386: snap_remove snap 34 2024-09-16T04:42:53.611 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:42:53.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:42:53.614 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:53.610+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:53.716 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:53.711+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:53.733 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:53.733 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:53.733 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:53.733 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:53.733 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:53.747 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:42:53.816 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:53.810+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:53.917 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:53.911+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4356.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.525316439999997, 2024-09-16T04:42:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31207237399999999, 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4356", 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.59:0/2607617674", 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398126+0000", 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0999999999999998e-06 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398137+0000", 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0253e-05 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398158+0000", 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1401999999999999e-05 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398176+0000", 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8552000000000001e-05 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597014+0000", 2024-09-16T04:42:53.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19883803 2024-09-16T04:42:53.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:53.933 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597043+0000", 2024-09-16T04:42:53.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8683999999999998e-05 2024-09-16T04:42:53.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.604764+0000", 2024-09-16T04:42:53.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0077206829999999999 2024-09-16T04:42:53.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605002+0000", 2024-09-16T04:42:53.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023771400000000001 2024-09-16T04:42:53.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 15", 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605234+0000", 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023226099999999999 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.606579+0000", 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013450650000000001 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710048+0000", 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103468869 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710127+0000", 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9491000000000002e-05 2024-09-16T04:42:53.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.938 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710197+0000", 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9270000000000006e-05 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:190 3.8s0 3:16f2ba44:::smithi16530038-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 65.823332262999998, 2024-09-16T04:42:53.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70466111799999998, 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 190 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:53.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100109+0000", 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7300000000000004e-07 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100116+0000", 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3099999999999997e-06 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100128+0000", 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2262000000000001e-05 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100137+0000", 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:42:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480895+0000", 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.38075793800000002 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480902+0000", 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0700000000000001e-06 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582260+0000", 2024-09-16T04:42:53.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101357841 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582314+0000", 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4175000000000002e-05 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804615+0000", 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.22230154099999999 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804681+0000", 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5356000000000002e-05 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804769+0000", 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8782000000000005e-05 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 61.514067476999998, 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.311529638, 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.946 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409375+0000", 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8899999999999999e-06 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409439+0000", 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4428e-05 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409480+0000", 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0454999999999998e-05 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409505+0000", 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4620000000000001e-05 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619957+0000", 2024-09-16T04:42:53.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21045218900000001 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619979+0000", 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1866e-05 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720830+0000", 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100851396 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.949 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720903+0000", 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2793999999999995e-05 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=31, reqid=client.4728.0:186, at_version=42'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 61.508782601999997, 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.306308988, 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.950 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414660+0000", 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414693+0000", 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414716+0000", 2024-09-16T04:42:53.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3356000000000001e-05 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414734+0000", 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7787000000000001e-05 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621156+0000", 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.206422364 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621180+0000", 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3589999999999999e-05 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720911+0000", 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099731400999999997 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720967+0000", 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5841000000000003e-05 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.954 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 56.559061757999999, 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30496093299999999, 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:53.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364381+0000", 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.663e-06 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.956 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364619+0000", 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023814999999999999 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364642+0000", 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2875999999999999e-05 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364655+0000", 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3849000000000001e-05 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.957 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.667942+0000", 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30328671699999998 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.669340+0000", 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013976780000000001 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 60/32 ECSubWrite(tid=123, reqid=osd.11.0:123, at_version=60'50, trim_to=0'0, roll_forward_to=59'48))", 2024-09-16T04:42:53.958 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 43.434957955000002, 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30259428900000002, 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.959 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488483+0000", 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5400000000000002e-07 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488522+0000", 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9271999999999999e-05 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.488552+0000", 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9745999999999999e-05 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.588697+0000", 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100144495 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.789105+0000", 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20040802099999999 2024-09-16T04:42:53.961 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.789120+0000", 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5818999999999999e-05 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.791010+0000", 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001889944 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:10.791077+0000", 2024-09-16T04:42:53.962 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6637999999999995e-05 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:523 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.690319348999999, 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41544228799999999, 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:53.963 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 523 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:42:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9069999999999999e-06 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233135+0000", 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1303e-05 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233167+0000", 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1415999999999999e-05 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233183+0000", 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6495999999999999e-05 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.239234+0000", 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006050153 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.966 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.239492+0000", 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000258872 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.648564+0000", 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40907114100000003 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:524 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.690230507999999, 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41687002699999998, 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 524 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.968 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233211+0000", 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0100000000000001e-06 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233233+0000", 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1647000000000001e-05 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233246+0000", 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.332e-05 2024-09-16T04:42:53.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233256+0000", 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7319999999999993e-06 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.240876+0000", 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076197690000000002 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241108+0000", 2024-09-16T04:42:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000232103 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.650080+0000", 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40897244599999999 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:525 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:42:53.972 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.690167604999999, 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41781336400000002, 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 525 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.973 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233274+0000", 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1999999999999998e-07 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.974 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233279+0000", 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6799999999999998e-06 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233293+0000", 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3169e-05 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233303+0000", 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0540000000000001e-05 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.975 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.343879+0000", 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11057620899999999 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344065+0000", 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000186083 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.651086+0000", 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30702086299999998 2024-09-16T04:42:53.976 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.17s0 65/53 ECSubRead(tid=25, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.681804364999998, 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30542288899999998, 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.977 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241846+0000", 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021020300000000001 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.979 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241873+0000", 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6367e-05 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.546861+0000", 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30498862300000001 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.547059+0000", 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197696 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.980 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 64/53 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.674506174000001, 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.39834076099999999, 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:42:53.981 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248935+0000", 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.982 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249102+0000", 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166933 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249127+0000", 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249156+0000", 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8572999999999999e-05 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647206+0000", 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.39805037799999998 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647275+0000", 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8837999999999994e-05 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 65/53 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.581598659999997, 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30554653199999998, 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341843+0000", 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.533e-06 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342145+0000", 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00030105099999999999 2024-09-16T04:42:53.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342175+0000", 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0159e-05 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342192+0000", 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7003e-05 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647333+0000", 2024-09-16T04:42:53.987 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30514099700000002 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647388+0000", 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5788999999999997e-05 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.17s0 65/53 ECSubRead(tid=26, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 38.578966002000001, 2024-09-16T04:42:53.988 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30318800499999998, 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.989 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344537+0000", 2024-09-16T04:42:53.990 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2811999999999994e-05 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344572+0000", 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4798999999999997e-05 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647490+0000", 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30291773100000002 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.991 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647663+0000", 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017266300000000001 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPushReply(3.1ds0 74/70 [PushReplyOp(3:bfd7179d:::smithi16530038-33:c)])", 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.424202842, 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30809602800000002, 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.992 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.993 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1399999999999997e-07 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499244+0000", 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8300000000000001e-06 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.994 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499262+0000", 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7346e-05 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499275+0000", 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3376e-05 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:53.996 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807219+0000", 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30794440000000001 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807334+0000", 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000114462 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPushReply(3.1ds0 74/70 [PushReplyOp(3:bfd7179d:::smithi16530038-33:c)])", 2024-09-16T04:42:53.997 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.424012962999999, 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30806645799999999, 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:53.998 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.46e-07 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499433+0000", 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5140000000000003e-06 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:53.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499446+0000", 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2918999999999999e-05 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499456+0000", 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0093e-05 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807424+0000", 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30796800600000002 2024-09-16T04:42:54.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807494+0000", 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0480000000000003e-05 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPushReply(3.1ds0 74/70 [PushReplyOp(3:bfd7179d:::smithi16530038-33:c)])", 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.500211+0000", 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.423229531, 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30692697899999999, 2024-09-16T04:42:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500211+0000", 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500211+0000", 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500212+0000", 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.13e-07 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500217+0000", 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5799999999999999e-06 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500230+0000", 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3200000000000001e-05 2024-09-16T04:42:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500248+0000", 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7402999999999999e-05 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807054+0000", 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30680581000000001 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807138+0000", 2024-09-16T04:42:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4373e-05 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:721 3.1ds0 3:bc0e3a40:::smithi16530038-18:head [append 0~737280] snapc 16=[16,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:31.461810+0000", 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 22.461630645, 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61923460500000005, 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 721 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.461810+0000", 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.461810+0000", 2024-09-16T04:42:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.461813+0000", 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.123e-06 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.463204+0000", 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013910960000000001 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.463242+0000", 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8083000000000001e-05 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.563407+0000", 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100164278 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.563612+0000", 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020560500000000001 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.563630+0000", 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8343e-05 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.865484+0000", 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30185366200000002 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.865675+0000", 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019052500000000001 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.868329+0000", 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026546040000000001 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.874291+0000", 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0059619089999999996 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:32.080892+0000", 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.206600961 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:32.081045+0000", 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000152416 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:742 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:35.444909+0000", 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.478531807, 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40172697000000002, 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 742 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444909+0000", 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444909+0000", 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444912+0000", 2024-09-16T04:42:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7659999999999999e-06 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444925+0000", 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3196999999999999e-05 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444958+0000", 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3160000000000001e-05 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444974+0000", 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6447e-05 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445141+0000", 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016653100000000001 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445164+0000", 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3499999999999999e-05 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.543056+0000", 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097891299000000001 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.543207+0000", 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015134299999999999 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.846636+0000", 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30342872700000001 2024-09-16T04:42:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:743 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:35.445011+0000", 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.478429857999998, 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40293953599999999, 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 743 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445011+0000", 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445011+0000", 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445012+0000", 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.283e-06 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445020+0000", 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2029999999999992e-06 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445044+0000", 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3354e-05 2024-09-16T04:42:54.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445060+0000", 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5973000000000001e-05 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445286+0000", 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022647100000000001 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445297+0000", 2024-09-16T04:42:54.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.097e-05 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.543835+0000", 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.098537793999999998 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.544002+0000", 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000167248 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.847950+0000", 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394823999999998 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:744 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:35.445135+0000", 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 18.478305886000001, 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.403995469, 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:54.022 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 744 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445135+0000", 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445135+0000", 2024-09-16T04:42:54.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445137+0000", 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445147+0000", 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.058e-05 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445166+0000", 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9110000000000002e-05 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.545261+0000", 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100094422 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.545385+0000", 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012448200000000001 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.545629+0000", 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024355700000000001 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.849130+0000", 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30350133800000001 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.026 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:54.028 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:42:54.029 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:54.022+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:54.129 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:54.123+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.11s3 103/102 ECSubRead(tid=37, to_read={3:8c8bcb2a:::smithi16530038-46:head=262144,262144,0}, subchunks={3:8c8bcb2a:::smithi16530038-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:54.090466+0000", 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.067945244000000002, 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.068002121999999998, 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.090466+0000", 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.090466+0000", 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.090468+0000", 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:42:54.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.090488+0000", 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0262999999999999e-05 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.090521+0000", 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2570000000000002e-05 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.090537+0000", 2024-09-16T04:42:54.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6586999999999999e-05 2024-09-16T04:42:54.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.162 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.162 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-09-16T04:42:54.162 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 1 2024-09-16T04:42:54.162 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:54.172 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:42:54.230 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:54.226+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.6s3 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.3s3 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.23s3 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.22s4 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.21s3 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.9s4 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.20s2 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.2s4 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.1fs1 2024-09-16T04:42:54.254 INFO:teuthology.orchestra.run.smithi165.stdout:3.19s4 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.16s1 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.8s5 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.14s2 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.12s6 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.11s2 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.5s0 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.1ds1 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.15s6 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.10s6 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:3.0s2 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:2.7 2024-09-16T04:42:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:2.3 2024-09-16T04:42:54.331 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:54.327+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:54.340 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:54.341 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:54.355 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:42:54.432 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:54.427+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:54.476 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:54.476 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:54.476 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:54.476 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:54.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.476 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=11, reqid=client.4728.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 68.387090348000001, 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20239900399999999, 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087079+0000", 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5730000000000002e-06 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087392+0000", 2024-09-16T04:42:54.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000313164 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087425+0000", 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2768999999999997e-05 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187523+0000", 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100097504 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187687+0000", 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164537 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187711+0000", 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4209000000000001e-05 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289394+0000", 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10168234399999999 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289476+0000", 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1903999999999994e-05 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.as3 36/32 ECSubRead(tid=13, to_read={3:5619390f:::smithi16530038-41:head=0,4096,0}, subchunks={3:5619390f:::smithi16530038-41:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 68.184593614999997, 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20090377300000001, 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.481 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289573+0000", 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289575+0000", 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.717e-06 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289598+0000", 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3326e-05 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289642+0000", 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3174999999999999e-05 2024-09-16T04:42:54.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.389821+0000", 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100179457 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490099+0000", 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10027802299999999 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.490477+0000", 2024-09-16T04:42:54.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037807500000000001 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 38/32 ECSubWrite(tid=18, reqid=client.4728.0:194, at_version=38'11, trim_to=0'0, roll_forward_to=36'9))", 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 65.986818143999997, 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202828813, 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:42:54.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487637+0000", 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00028784999999999998 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487673+0000", 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5542e-05 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.587811+0000", 2024-09-16T04:42:54.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100137777 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.688010+0000", 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10019921599999999 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.688025+0000", 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4955e-05 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.488 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.690131+0000", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002106442 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.690178+0000", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6381000000000002e-05 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 44/41 ECSubWrite(tid=55, reqid=osd.4.0:55, at_version=44'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:54.489 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 60.260855026000002, 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.296361333, 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2e-07 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213370+0000", 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7278000000000003e-05 2024-09-16T04:42:54.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213391+0000", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1696999999999999e-05 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213413+0000", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1104999999999999e-05 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.407449+0000", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19403645 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.407462+0000", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3315e-05 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509615+0000", 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102152944 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509673+0000", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8023999999999999e-05 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 44/41 ECSubWrite(tid=59, reqid=client.4728.0:299, at_version=44'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 60.251075559, 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.28930476599999999, 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.494 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223094+0000", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6359999999999998e-06 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223754+0000", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00066019600000000005 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223785+0000", 2024-09-16T04:42:54.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1165999999999999e-05 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223806+0000", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0134999999999999e-05 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.508219+0000", 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.28441328900000001 2024-09-16T04:42:54.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.508234+0000", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5213e-05 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.512334+0000", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041000660000000003 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.512396+0000", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2064999999999996e-05 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 58/37 ECSubWrite(tid=117, reqid=client.4728.0:455, at_version=58'27, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.403858+0000", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.070309092000002, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.227776374, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.499 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403858+0000", 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403858+0000", 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403859+0000", 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.023e-06 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403890+0000", 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1013000000000003e-05 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403917+0000", 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6965999999999999e-05 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403934+0000", 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7085999999999999e-05 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.629673+0000", 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.22573882300000001 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.629696+0000", 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2745999999999999e-05 2024-09-16T04:42:54.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.631574+0000", 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001878242 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.631634+0000", 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0475e-05 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 58/53 ECSubWrite(tid=60, reqid=osd.3.0:60, at_version=58'32, trim_to=0'0, roll_forward_to=57'30))", 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.423074+0000", 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.051093078000001, 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20255648300000001, 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423074+0000", 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423074+0000", 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423076+0000", 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.793e-06 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423130+0000", 2024-09-16T04:42:54.505 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4191999999999999e-05 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423167+0000", 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6770000000000002e-05 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423190+0000", 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3629e-05 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.506 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.523695+0000", 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100505223 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.523724+0000", 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8276000000000002e-05 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.625547+0000", 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101823598 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.625630+0000", 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3002000000000003e-05 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 58/53 ECSubWrite(tid=61, reqid=osd.3.0:61, at_version=58'34, trim_to=0'0, roll_forward_to=57'30))", 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.423858+0000", 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.050308776999998, 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20264642399999999, 2024-09-16T04:42:54.508 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423858+0000", 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423858+0000", 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.509 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423860+0000", 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.747e-06 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423929+0000", 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9251999999999994e-05 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423965+0000", 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5478999999999997e-05 2024-09-16T04:42:54.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423988+0000", 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3048999999999999e-05 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.624695+0000", 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20070756300000001 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.624728+0000", 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2353e-05 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.626430+0000", 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001702606 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.626505+0000", 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4375000000000005e-05 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.512 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 59/37 ECSubWrite(tid=129, reqid=osd.4.0:126, at_version=0'0, trim_to=0'0, roll_forward_to=59'29))", 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:09.452509+0000", 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 45.021657996999998, 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20064473199999999, 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.452509+0000", 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.452509+0000", 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.452511+0000", 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7999999999999999e-06 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.452532+0000", 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1725999999999998e-05 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.452564+0000", 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1083e-05 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.452587+0000", 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3255999999999999e-05 2024-09-16T04:42:54.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.453319+0000", 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00073179600000000005 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.453345+0000", 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6157e-05 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.516 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.653065+0000", 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19972071999999999 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:09.653154+0000", 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8194000000000003e-05 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 70/67 ECSubWrite(tid=82, reqid=osd.2.0:82, at_version=70'56, trim_to=0'0, roll_forward_to=69'54))", 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:20.362623+0000", 2024-09-16T04:42:54.517 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 34.111543566999998, 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20271673900000001, 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362623+0000", 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362623+0000", 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362624+0000", 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6300000000000005e-07 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362649+0000", 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4700000000000001e-05 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362659+0000", 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0521e-05 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.462788+0000", 2024-09-16T04:42:54.520 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100128973 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.463788+0000", 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099988700000000008 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.463802+0000", 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4321999999999999e-05 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.521 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.565297+0000", 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101494468 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.565340+0000", 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3305e-05 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.522 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 70/53 ECSubWrite(tid=84, reqid=osd.3.0:82, at_version=0'0, trim_to=0'0, roll_forward_to=70'46))", 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:20.568258+0000", 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 33.905908443999998, 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201006248, 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.568258+0000", 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.568258+0000", 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.568260+0000", 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.536e-06 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.568285+0000", 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4944e-05 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.568315+0000", 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9794999999999998e-05 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.568337+0000", 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2079999999999999e-05 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.668556+0000", 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10021870400000001 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.668576+0000", 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0020000000000001e-05 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.769215+0000", 2024-09-16T04:42:54.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100639855 2024-09-16T04:42:54.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.527 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.527 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.769265+0000", 2024-09-16T04:42:54.527 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9314000000000003e-05 2024-09-16T04:42:54.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.527 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.528 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.528 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.528 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.528 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 80/67 ECSubWrite(tid=116, reqid=osd.2.0:114, at_version=0'0, trim_to=0'0, roll_forward_to=80'75))", 2024-09-16T04:42:54.528 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.575809+0000", 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.898358295000001, 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20150364200000001, 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575809+0000", 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.529 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575809+0000", 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575811+0000", 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8640000000000001e-06 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575833+0000", 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2599000000000002e-05 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.530 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575863+0000", 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9484000000000001e-05 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.675964+0000", 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100101125 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676136+0000", 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017185099999999999 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676166+0000", 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0573e-05 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.777230+0000", 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101063783 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.777312+0000", 2024-09-16T04:42:54.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2362999999999997e-05 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 80/53 ECSubWrite(tid=130, reqid=osd.3.0:130, at_version=80'69, trim_to=0'0, roll_forward_to=79'67))", 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.676168+0000", 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.797998989, 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20210377600000001, 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.533 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676168+0000", 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676168+0000", 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.534 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676169+0000", 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0139999999999999e-06 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676206+0000", 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6566000000000002e-05 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676227+0000", 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1140000000000001e-05 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776333+0000", 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100106442 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876610+0000", 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10027724 2024-09-16T04:42:54.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876627+0000", 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6416e-05 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.878207+0000", 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015805719999999999 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.878272+0000", 2024-09-16T04:42:54.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4386000000000003e-05 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 80/67 ECSubWrite(tid=131, reqid=osd.3.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=80'51))", 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.776388+0000", 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.697778952, 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20107434499999999, 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.538 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776388+0000", 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776388+0000", 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776389+0000", 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0640000000000001e-06 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776406+0000", 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7309999999999999e-05 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776425+0000", 2024-09-16T04:42:54.540 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8790000000000001e-05 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776437+0000", 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2283000000000001e-05 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876715+0000", 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10027707299999999 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876731+0000", 2024-09-16T04:42:54.541 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6460000000000002e-05 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.977340+0000", 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100609109 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.977462+0000", 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000122256 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 80/53 ECSubWrite(tid=132, reqid=osd.3.0:130, at_version=0'0, trim_to=0'0, roll_forward_to=80'69))", 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.907366+0000", 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 23.566801419000001, 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201024537, 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907366+0000", 2024-09-16T04:42:54.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907366+0000", 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907367+0000", 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.933e-06 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907393+0000", 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.584e-05 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907433+0000", 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9403000000000002e-05 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907455+0000", 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2643e-05 2024-09-16T04:42:54.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.007692+0000", 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10023647400000001 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.007708+0000", 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6133e-05 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.108290+0000", 2024-09-16T04:42:54.546 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100581795 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.108390+0000", 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010031600000000001 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 90/67 ECSubWrite(tid=146, reqid=osd.2.0:144, at_version=0'0, trim_to=0'0, roll_forward_to=90'84))", 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:41.611525+0000", 2024-09-16T04:42:54.547 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.862641639, 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20182486499999999, 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611525+0000", 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611525+0000", 2024-09-16T04:42:54.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611526+0000", 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8599999999999999e-07 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.549 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611540+0000", 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4209999999999999e-05 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611558+0000", 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8383e-05 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.711659+0000", 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100100222 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.550 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.811951+0000", 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10029222 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.811984+0000", 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3367000000000001e-05 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.813285+0000", 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013007680000000001 2024-09-16T04:42:54.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.813350+0000", 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5308999999999994e-05 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 92/67 ECSubWrite(tid=203, reqid=client.4728.0:827, at_version=92'51, trim_to=0'0, roll_forward_to=90'49))", 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:42.724950+0000", 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.749216542999999, 2024-09-16T04:42:54.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20312651500000001, 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.724950+0000", 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.724950+0000", 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.724952+0000", 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9460000000000001e-06 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725005+0000", 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2482999999999999e-05 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725041+0000", 2024-09-16T04:42:54.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5743e-05 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725064+0000", 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3796999999999999e-05 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.825306+0000", 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100241371 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.825333+0000", 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7716999999999999e-05 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.927997+0000", 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10266338799999999 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.928077+0000", 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0069999999999997e-05 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.556 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 92/67 ECSubWrite(tid=204, reqid=osd.4.0:204, at_version=92'53, trim_to=0'0, roll_forward_to=90'49))", 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.749051378000001, 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20378613500000001, 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9999999999999997e-07 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725162+0000", 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5952999999999999e-05 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725179+0000", 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7073000000000001e-05 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.825284+0000", 2024-09-16T04:42:54.559 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100105179 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.927398+0000", 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10211313800000001 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.927415+0000", 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7166000000000001e-05 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.560 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.928821+0000", 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001406247 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.928902+0000", 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0679000000000005e-05 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPush(3.19s3 95/58 [PushOp(3:9de467d5:::smithi16530038-27:7, version: 50'27, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:9de467d5:::smithi16530038-27:7@50'27, size: 1294336, copy_subset: [], clone_subset: {}, snapset: 11=[]:{4: [3], 7: [7], 11: [11, d, b]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:54.561 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:46.634171+0000", 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.8399957139999996, 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20206529000000001, 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634171+0000", 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634171+0000", 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634173+0000", 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.193e-06 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634920+0000", 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00074700600000000004 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634992+0000", 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1847000000000005e-05 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.735128+0000", 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100136155 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.835418+0000", 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100289403 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.836237+0000", 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000818686 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 97/67 ECSubWrite(tid=216, reqid=osd.4.0:216, at_version=97'57, trim_to=0'0, roll_forward_to=93'55))", 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:47.954069+0000", 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.5200979549999998, 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20286156699999999, 2024-09-16T04:42:54.565 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954069+0000", 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954069+0000", 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954071+0000", 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.843e-06 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954112+0000", 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0893000000000003e-05 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954138+0000", 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6432999999999999e-05 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.054245+0000", 2024-09-16T04:42:54.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100106473 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.154543+0000", 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10029845499999999 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.154559+0000", 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6276000000000001e-05 2024-09-16T04:42:54.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.156854+0000", 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022942919999999999 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.156931+0000", 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6902000000000003e-05 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.571 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.571 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:54.572 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:42:54.573 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:54.571+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:54.589 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:42:54.595 INFO:tasks.rados.rados.0.smithi165.stdout:386: done (5 left) 2024-09-16T04:42:54.595 INFO:tasks.rados.rados.0.smithi165.stdout:387: rollback oid 13 current snap is 35 2024-09-16T04:42:54.595 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 27 2024-09-16T04:42:54.595 INFO:tasks.rados.rados.0.smithi165.stdout:382: finishing write tid 6 to smithi16530038-9 2024-09-16T04:42:54.595 INFO:tasks.rados.rados.0.smithi165.stdout:382: oid 9 updating version 0 to 105 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:382: oid 9 version 105 is already newer than 104 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:382: oid 9 version 105 is already newer than 103 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:382: oid 9 version 105 is already newer than 102 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:382: oid 9 version 105 is already newer than 101 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 105 (ObjNum 140 snap 34 seq_num 140) dirty exists 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:382: left oid 9 (ObjNum 140 snap 34 seq_num 140) 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 113 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:385: finishing copy_from to smithi16530038-26 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:385: got expected ENOENT (src dne) 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:385: finishing copy_from racing read to smithi16530038-26 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:382: done (5 left) 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:383: done (4 left) 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:385: done (3 left) 2024-09-16T04:42:54.596 INFO:tasks.rados.rados.0.smithi165.stdout:388: read oid 33 snap -1 2024-09-16T04:42:54.597 INFO:tasks.rados.rados.0.smithi165.stdout:388: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:42:54.597 INFO:tasks.rados.rados.0.smithi165.stdout:389: read oid 29 snap -1 2024-09-16T04:42:54.597 INFO:tasks.rados.rados.0.smithi165.stdout:389: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:54.597 INFO:tasks.rados.rados.0.smithi165.stdout:390: delete oid 21 current snap is 35 2024-09-16T04:42:54.673 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:54.667+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.fs3 104/67 ECSubRead(tid=282, to_read={3:f1175b03:::smithi16530038-29:head=0,57344,0}, subchunks={3:f1175b03:::smithi16530038-29:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:54.705855+0000", 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.003506913, 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035719520000000002, 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "no flag points reached", 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.705855+0000", 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.705855+0000", 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.711 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.705856+0000", 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8500000000000001e-06 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.705878+0000", 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.109e-05 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.705922+0000", 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4799000000000003e-05 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.712 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:54.713 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.713 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:54.713 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-09-16T04:42:54.713 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 1 2024-09-16T04:42:54.713 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:54.713 INFO:tasks.rados.rados.0.smithi165.stdout:377: finishing rollback tid 0 to smithi16530038-46 2024-09-16T04:42:54.726 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:42:54.774 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:54.771+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:54.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-15 --journal-path /var/lib/ceph/osd/ceph-15/journal --op export --pgid 3.2s4 --file /var/log/ceph/exp.3.2s4.15 2024-09-16T04:42:54.813 INFO:tasks.rados.rados.0.smithi165.stdout:377: finishing rollback tid 1 to smithi16530038-46 2024-09-16T04:42:54.815 INFO:tasks.rados.rados.0.smithi165.stdout:377: finishing rollback tid 2 to smithi16530038-46 2024-09-16T04:42:54.815 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 50 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-16T04:42:54.852 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:54.852 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:42:54.852 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:42:54.852 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:42:54.852 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:54.866 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:42:54.911 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:54.907+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:54.917 INFO:tasks.rados.rados.0.smithi165.stdout:380: finishing copy_from to smithi16530038-14 2024-09-16T04:42:54.917 INFO:tasks.rados.rados.0.smithi165.stdout:380: got expected ENOENT (src dne) 2024-09-16T04:42:54.919 INFO:tasks.rados.rados.0.smithi165.stdout:380: finishing copy_from racing read to smithi16530038-14 2024-09-16T04:42:54.976 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:54.971+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:54.996 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:42:54.996 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:42:54.996 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:42:54.996 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.617319445000007, 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1755266710000001, 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:54.997 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.998 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374332+0000", 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8059999999999999e-06 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374345+0000", 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2529e-05 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:54.999 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374365+0000", 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0695000000000001e-05 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374383+0000", 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7569000000000001e-05 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374514+0000", 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013070099999999999 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374544+0000", 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0731000000000001e-05 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.343896+0000", 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.969351224 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.001 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344169+0000", 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027390100000000001 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344583+0000", 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041308600000000001 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.346047+0000", 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014642699999999999 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549695+0000", 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203648463 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549791+0000", 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6137999999999993e-05 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549856+0000", 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:42:55.003 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 82.370903265999999, 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51346886800000002, 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.004 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 111 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.005 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620747+0000", 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620757+0000", 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0316000000000001e-05 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620778+0000", 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:42:55.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620792+0000", 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4362000000000001e-05 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928510+0000", 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.307717717 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928629+0000", 2024-09-16T04:42:55.007 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011919 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928822+0000", 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019246199999999999 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930635+0000", 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00181292 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134065+0000", 2024-09-16T04:42:55.008 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20343018399999999 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134149+0000", 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4635000000000005e-05 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134214+0000", 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4973000000000003e-05 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 82.370832902999993, 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51366009000000001, 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 112 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.010 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620817+0000", 2024-09-16T04:42:55.011 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621035+0000", 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021731100000000001 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621061+0000", 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6744999999999999e-05 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.012 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621082+0000", 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1061999999999999e-05 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929460+0000", 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.308377346 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929595+0000", 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013512600000000001 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929798+0000", 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020277199999999999 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932465+0000", 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002667862 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134319+0000", 2024-09-16T04:42:55.014 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20185381599999999 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134407+0000", 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7894999999999998e-05 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134476+0000", 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8829000000000001e-05 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.015 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 82.370532960000006, 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51357863199999998, 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.016 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 113 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621118+0000", 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.689e-06 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621129+0000", 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621164+0000", 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5367999999999997e-05 2024-09-16T04:42:55.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621182+0000", 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8352e-05 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930689+0000", 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.309507015 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930800+0000", 2024-09-16T04:42:55.019 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000110871 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931026+0000", 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000226293 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.933569+0000", 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002542786 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.020 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134566+0000", 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200996272 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134640+0000", 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4412000000000005e-05 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134694+0000", 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4524000000000002e-05 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 82.370436213000005, 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51372108500000002, 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 114 2024-09-16T04:42:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621214+0000", 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621225+0000", 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1348999999999999e-05 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621245+0000", 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9966000000000002e-05 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621259+0000", 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4102e-05 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931642+0000", 2024-09-16T04:42:55.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31038274300000002 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931776+0000", 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133552 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931973+0000", 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197262 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.033893+0000", 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101919751 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134774+0000", 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10088169600000001 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134847+0000", 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2309000000000002e-05 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134934+0000", 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6924999999999999e-05 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 82.370364404, 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51387291300000004, 2024-09-16T04:42:55.028 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 115 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621286+0000", 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.26e-06 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621492+0000", 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020643500000000001 2024-09-16T04:42:55.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621517+0000", 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621534+0000", 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6515999999999999e-05 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932515+0000", 2024-09-16T04:42:55.031 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31098146799999998 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932631+0000", 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011636 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932814+0000", 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000182406 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134236+0000", 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201422618 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135029+0000", 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079253199999999998 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135102+0000", 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3286000000000003e-05 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135157+0000", 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5043000000000001e-05 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 82.370082585999995, 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.717172162, 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.034 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 116 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.035 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621568+0000", 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621606+0000", 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8167999999999998e-05 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621628+0000", 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1742e-05 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621643+0000", 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5696e-05 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235313+0000", 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61366968300000002 2024-09-16T04:42:55.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235449+0000", 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136036 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235657+0000", 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207405 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237049+0000", 2024-09-16T04:42:55.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001392552 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338582+0000", 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101532626 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338670+0000", 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7888999999999998e-05 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.039 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338738+0000", 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8769000000000005e-05 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 82.369974596000006, 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.81671015999999996, 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.040 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 117 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621676+0000", 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4440000000000001e-06 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621686+0000", 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0845e-05 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621706+0000", 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9168999999999999e-05 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621723+0000", 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7538999999999999e-05 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236241+0000", 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61451815600000004 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236372+0000", 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013062300000000001 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236574+0000", 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020165600000000001 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237570+0000", 2024-09-16T04:42:55.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099642999999999997 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438224+0000", 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20065369 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438317+0000", 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3624999999999999e-05 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.045 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438384+0000", 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6983000000000006e-05 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 80.696900619000004, 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.508075677, 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.046 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 138 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294750+0000", 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294760+0000", 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294779+0000", 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8941999999999999e-05 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294794+0000", 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5038999999999999e-05 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405306+0000", 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11051159300000001 2024-09-16T04:42:55.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405426+0000", 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011977 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405611+0000", 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000185217 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709561+0000", 2024-09-16T04:42:55.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394976699999998 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802549+0000", 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092988137999999998 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802707+0000", 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015861799999999999 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.051 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802824+0000", 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116517 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 80.696830496000004, 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60756662299999997, 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.052 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 139 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294820+0000", 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4330000000000001e-06 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294830+0000", 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0023e-05 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294850+0000", 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0622e-05 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294864+0000", 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4042e-05 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406110+0000", 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11124526 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406222+0000", 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111871 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406398+0000", 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017598000000000001 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709577+0000", 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30317971500000002 2024-09-16T04:42:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902101+0000", 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19252423499999999 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902267+0000", 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016537499999999999 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.057 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902385+0000", 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011806700000000001 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 80.696738367999998, 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60781196900000001, 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.058 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 140 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:42:55.059 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294912+0000", 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.407e-06 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295040+0000", 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012836600000000001 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295062+0000", 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1682e-05 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295083+0000", 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1109000000000001e-05 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507008+0000", 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21192486699999999 2024-09-16T04:42:55.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507149+0000", 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141042 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507369+0000", 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022046500000000001 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710298+0000", 2024-09-16T04:42:55.062 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202928308 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902525+0000", 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.192226958 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902611+0000", 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6274999999999996e-05 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.063 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902722+0000", 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011149 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 80.696510786999994, 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70811914799999998, 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.064 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 141 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.065 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295140+0000", 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295152+0000", 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2053e-05 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295172+0000", 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0282e-05 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295191+0000", 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8759000000000001e-05 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508486+0000", 2024-09-16T04:42:55.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.213295545 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508630+0000", 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014419899999999999 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508824+0000", 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019358600000000001 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710318+0000", 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20149381 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.002981+0000", 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.29266302500000002 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003144+0000", 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016330500000000001 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003257+0000", 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113064 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.070 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 79.032054059000004, 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51415646199999998, 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.071 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 159 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.072 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959597+0000", 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9190000000000002e-06 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959609+0000", 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959629+0000", 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0035e-05 2024-09-16T04:42:55.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959645+0000", 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5786000000000001e-05 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167529+0000", 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.207883768 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.074 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167762+0000", 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023295499999999999 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.168134+0000", 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037181099999999999 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.270830+0000", 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102696683 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473585+0000", 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20275470700000001 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473677+0000", 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1772000000000002e-05 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473751+0000", 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4228999999999997e-05 2024-09-16T04:42:55.076 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 79.031973296999993, 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51420228400000001, 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.077 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 160 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959677+0000", 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959687+0000", 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959711+0000", 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3811999999999998e-05 2024-09-16T04:42:55.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959727+0000", 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169193+0000", 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209466241 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169340+0000", 2024-09-16T04:42:55.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000147279 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169545+0000", 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000205167 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374117+0000", 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20457175999999999 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473763+0000", 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099646306000000004 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473828+0000", 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4046000000000003e-05 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473878+0000", 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0241000000000002e-05 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.082 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 79.031896017999998, 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51424688600000001, 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.083 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 161 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.084 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959754+0000", 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959920+0000", 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166088 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959944+0000", 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959963+0000", 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9176000000000001e-05 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371049+0000", 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41108644500000002 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371185+0000", 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013619899999999999 2024-09-16T04:42:55.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371395+0000", 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020912 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.377708+0000", 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006313173 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473891+0000", 2024-09-16T04:42:55.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.096182904 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473949+0000", 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8016999999999997e-05 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474000+0000", 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.088 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 79.031580261000002, 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51552797100000003, 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.089 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 162 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960070+0000", 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960077+0000", 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6830000000000008e-06 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960091+0000", 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3653e-05 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960101+0000", 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7629999999999995e-06 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372554+0000", 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41245356599999999 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372785+0000", 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023023100000000001 2024-09-16T04:42:55.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.373156+0000", 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000371601 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474010+0000", 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100854291 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475328+0000", 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001317304 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475477+0000", 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014954899999999999 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475596+0000", 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011913699999999999 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.094 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 79.031532033000005, 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51788103100000005, 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.095 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 163 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960118+0000", 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960124+0000", 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960134+0000", 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0115999999999999e-05 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960144+0000", 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374237+0000", 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41409377600000002 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374462+0000", 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022478600000000001 2024-09-16T04:42:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374797+0000", 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000334586 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474018+0000", 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099220947000000004 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477813+0000", 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003795171 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477921+0000", 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010822100000000001 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477998+0000", 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6662999999999994e-05 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 79.031492818999993, 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51822496799999995, 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 164 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960157+0000", 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7399999999999991e-07 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960241+0000", 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3677999999999994e-05 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960256+0000", 2024-09-16T04:42:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5206e-05 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960267+0000", 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1372999999999999e-05 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375650+0000", 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41538323100000002 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375885+0000", 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000234758 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.376293+0000", 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040777599999999999 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474024+0000", 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097731440000000003 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478113+0000", 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004088225 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478261+0000", 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014825 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478381+0000", 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012005700000000001 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 79.031366746000003, 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51862660500000002, 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 165 2024-09-16T04:42:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960283+0000", 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960289+0000", 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.49e-06 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960301+0000", 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2585999999999999e-05 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960309+0000", 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472635+0000", 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51232596699999999 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472771+0000", 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136243 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473033+0000", 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026155699999999999 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474539+0000", 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015065 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478656+0000", 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041164319999999997 2024-09-16T04:42:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478793+0000", 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013757299999999999 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478909+0000", 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011530400000000001 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:153 3.3s0 3:c31e3ec6:::smithi16530038-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 68.702067614000001, 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.511692229, 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 153 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:42:55.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289583+0000", 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.04e-06 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289590+0000", 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4899999999999997e-06 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289610+0000", 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9814999999999999e-05 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289625+0000", 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5553000000000002e-05 2024-09-16T04:42:55.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.296844+0000", 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072185369999999997 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297045+0000", 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020125199999999999 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599740+0000", 2024-09-16T04:42:55.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.302695037 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599866+0000", 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012644500000000001 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.801273+0000", 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20140706 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:42:55.118 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:42:55.118 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:42:55.119 INFO:tasks.rados.rados.0.smithi165.stdout:377: done (5 left) 2024-09-16T04:42:55.119 INFO:tasks.rados.rados.0.smithi165.stdout:380: done (4 left) 2024-09-16T04:42:55.119 INFO:tasks.rados.rados.0.smithi165.stdout:390: done (3 left) 2024-09-16T04:42:55.119 INFO:tasks.rados.rados.0.smithi165.stdout:391: rollback oid 46 current snap is 35 2024-09-16T04:42:55.119 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 33 2024-09-16T04:42:55.120 INFO:tasks.rados.rados.0.smithi165.stdout:392: snap_remove snap 24 2024-09-16T04:42:55.120 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:55.115+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:55.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:42:55.220 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:55.215+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:55.250 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:55.250 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:55.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.250 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 104/97 ECSubWriteReply(tid=201, last_complete=104'98, committed=1, applied=1))", 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.111298+0000", 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.138239892, 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13827641199999999, 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111298+0000", 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111298+0000", 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111299+0000", 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.9699999999999994e-07 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111308+0000", 2024-09-16T04:42:55.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2029999999999992e-06 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111328+0000", 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0930000000000001e-05 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111340+0000", 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1566000000000001e-05 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.1es0 104/97 ECSubWriteReply(tid=202, last_complete=104'100, committed=1, applied=1))", 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.111358+0000", 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.13817977000000001, 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.138330906, 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111358+0000", 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111358+0000", 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111359+0000", 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8400000000000005e-07 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111362+0000", 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5429999999999998e-06 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111375+0000", 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2607e-05 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.111384+0000", 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 2 2024-09-16T04:42:55.257 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:55.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:42:55.321 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:55.314+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:55.388 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:55.388 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:55.388 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:55.388 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:55.388 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:55.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:42:55.446 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:55.443+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:55.523 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:55.518+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:108 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 70.161617578999994, 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60644887800000002, 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.529 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 108 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362769+0000", 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.364e-06 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362776+0000", 2024-09-16T04:42:55.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9789999999999996e-06 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362795+0000", 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8569000000000001e-05 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362811+0000", 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6084999999999999e-05 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369499+0000", 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066885180000000001 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369745+0000", 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024616900000000002 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669288+0000", 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29954281100000002 2024-09-16T04:42:55.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669425+0000", 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013708200000000001 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.969217+0000", 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29979130100000001 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:507 3.es0 3:7777e846:::smithi16530038-17:head [copy-from ver 21] snapc 10=[10,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.252562640999997, 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.526543015, 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.535 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 507 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271824+0000", 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7990000000000001e-06 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271832+0000", 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271852+0000", 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9194999999999998e-05 2024-09-16T04:42:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.371991+0000", 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10013957800000001 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372175+0000", 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000183325 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372198+0000", 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3306e-05 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472740+0000", 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100542016 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472902+0000", 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016220399999999999 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692422+0000", 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21951984699999999 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.796322+0000", 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10389983699999999 2024-09-16T04:42:55.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798260+0000", 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001938103 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798366+0000", 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010582900000000001 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.541 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:582 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 36.185016585, 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69089629399999997, 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.542 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 582 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339370+0000", 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339374+0000", 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0799999999999999e-06 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339387+0000", 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3464e-05 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339414+0000", 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7223e-05 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446804+0000", 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.107389545 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446957+0000", 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000153389 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727239+0000", 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28028220199999998 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727462+0000", 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022226099999999999 2024-09-16T04:42:55.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.030265+0000", 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280332199999999 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:613 3.es0 3:7777e846:::smithi16530038-17:head [append 0~589824] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.164166096999999, 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55445315100000003, 2024-09-16T04:42:55.547 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 613 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360223+0000", 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3730000000000001e-06 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361140+0000", 2024-09-16T04:42:55.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00091743300000000005 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361178+0000", 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7691999999999998e-05 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361197+0000", 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.891e-05 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361351+0000", 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015419300000000001 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361371+0000", 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9726000000000001e-05 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366548+0000", 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051778730000000004 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366714+0000", 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000165329 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.369378+0000", 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026643159999999999 2024-09-16T04:42:55.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.372406+0000", 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003027891 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914520+0000", 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54211425499999999 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914672+0000", 2024-09-16T04:42:55.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015216000000000001 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:614 3.es0 3:7777e846:::smithi16530038-17:head [append 0~540672] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.160193051, 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55075342599999999, 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.554 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 614 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364194+0000", 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.26e-06 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365041+0000", 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00084621499999999999 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365073+0000", 2024-09-16T04:42:55.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2719000000000003e-05 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465237+0000", 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100163671 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465392+0000", 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015528300000000001 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465730+0000", 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000338005 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.467540+0000", 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001809583 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.469612+0000", 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020719760000000001 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914847+0000", 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.44523504899999999 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914946+0000", 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8665000000000005e-05 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:617 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:42:55.560 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 30.957678921999999, 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71993709900000002, 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 617 2024-09-16T04:42:55.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566707+0000", 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566713+0000", 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.784e-06 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566725+0000", 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.173e-05 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566732+0000", 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6860000000000006e-06 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579439+0000", 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012706935000000001 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579759+0000", 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032009600000000002 2024-09-16T04:42:55.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074376+0000", 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49461618299999999 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074520+0000", 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014481800000000001 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.286643+0000", 2024-09-16T04:42:55.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212123017 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:696 3.10s0 3:0fc9379d:::smithi16530038-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 26.099676687999999, 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69507003199999995, 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.566 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 696 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424713+0000", 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5599999999999998e-06 2024-09-16T04:42:55.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424724+0000", 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0906e-05 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424732+0000", 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6499999999999996e-06 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502418+0000", 2024-09-16T04:42:55.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077686693000000001 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502530+0000", 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000111644 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917065+0000", 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41453470199999998 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917391+0000", 2024-09-16T04:42:55.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032674700000000001 2024-09-16T04:42:55.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.119779+0000", 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202387337 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:805 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.01932195, 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51150757400000002, 2024-09-16T04:42:55.572 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 805 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505067+0000", 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8139999999999998e-06 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505082+0000", 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4453e-05 2024-09-16T04:42:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505124+0000", 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2849999999999998e-05 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505148+0000", 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3062999999999999e-05 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611642+0000", 2024-09-16T04:42:55.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106494218 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611801+0000", 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000159338 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.613368+0000", 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001566462 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.711952+0000", 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098584630000000006 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016439+0000", 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30448657000000001 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016571+0000", 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013217599999999999 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:806 3.10s0 3:0fc9379d:::smithi16530038-35:head [rollback 11] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.019188621, 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89166837600000004, 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.578 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 806 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505198+0000", 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7239999999999999e-06 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505210+0000", 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1688999999999999e-05 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505232+0000", 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1903999999999999e-05 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505249+0000", 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7036000000000001e-05 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017365+0000", 2024-09-16T04:42:55.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.512115616 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017525+0000", 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016080200000000001 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.018648+0000", 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011226109999999999 2024-09-16T04:42:55.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021794+0000", 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031461660000000002 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396721+0000", 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.37492636899999998 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396865+0000", 2024-09-16T04:42:55.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014445899999999999 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:807 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.019107298, 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89196936699999996, 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.584 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 807 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505279+0000", 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505285+0000", 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0399999999999998e-06 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505308+0000", 2024-09-16T04:42:55.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2595999999999998e-05 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505323+0000", 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5342999999999999e-05 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019548+0000", 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51422475999999995 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019822+0000", 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027388100000000002 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021062+0000", 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001239614 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.122291+0000", 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101229427 2024-09-16T04:42:55.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397057+0000", 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27476627300000001 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397247+0000", 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000190096 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:809 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~540672] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.007092887000001, 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70063091600000005, 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 809 2024-09-16T04:42:55.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517297+0000", 2024-09-16T04:42:55.591 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8899999999999998e-06 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518377+0000", 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010801750000000001 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518421+0000", 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3114000000000001e-05 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.592 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518443+0000", 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2782e-05 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518619+0000", 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017519200000000001 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518647+0000", 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8053999999999999e-05 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698939+0000", 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180292338 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.699242+0000", 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000302911 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.702023+0000", 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027812560000000002 2024-09-16T04:42:55.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813606+0000", 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11158314900000001 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217721+0000", 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40411436000000001 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217923+0000", 2024-09-16T04:42:55.595 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000202695 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:810 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~720896] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.001087909000001, 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81676015000000002, 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.596 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 810 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:42:55.597 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523300+0000", 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1140000000000002e-06 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524411+0000", 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001110307 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.598 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524443+0000", 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2067000000000003e-05 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524458+0000", 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5223e-05 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524644+0000", 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018582900000000001 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.599 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524656+0000", 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1705999999999999e-05 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.704974+0000", 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180318379 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.705144+0000", 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017024499999999999 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.600 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.706863+0000", 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00171842 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813610+0000", 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10674731 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.339927+0000", 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52631677099999996 2024-09-16T04:42:55.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340058+0000", 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130779 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:811 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~786432] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.994435433, 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81051717700000003, 2024-09-16T04:42:55.602 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 811 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.603 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529953+0000", 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.824e-06 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531687+0000", 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017341699999999999 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531721+0000", 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3756e-05 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531744+0000", 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531894+0000", 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000150505 2024-09-16T04:42:55.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531914+0000", 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0137e-05 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808146+0000", 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.276231536 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808322+0000", 2024-09-16T04:42:55.606 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000175899 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.810020+0000", 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016987230000000001 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813806+0000", 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0037855419999999998 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340306+0000", 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52649966299999995 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340467+0000", 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016146800000000001 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:812 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~49152] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:55.608 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.992602836, 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80902035800000005, 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 812 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531784+0000", 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.796e-06 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531896+0000", 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011207 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531918+0000", 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1412000000000001e-05 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531934+0000", 2024-09-16T04:42:55.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6663999999999998e-05 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698290+0000", 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16635518999999999 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698310+0000", 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0587000000000001e-05 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811428+0000", 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11311787800000001 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811601+0000", 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017340500000000001 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.812624+0000", 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001022659 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815470+0000", 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002845752 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340654+0000", 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52518383899999999 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340803+0000", 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014910600000000001 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:813 3.22s0 3:456a3a59:::smithi16530038-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.992424345, 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80925618399999999, 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.615 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 813 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531962+0000", 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4339999999999999e-06 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531969+0000", 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7329999999999997e-06 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531987+0000", 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7467000000000001e-05 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632132+0000", 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10014582499999999 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813864+0000", 2024-09-16T04:42:55.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18173177200000001 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814024+0000", 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016021500000000001 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814762+0000", 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00073760799999999995 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.816600+0000", 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001837679 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341069+0000", 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52446895500000001 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341217+0000", 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000148496 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.620 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:814 3.22s0 3:456a3a59:::smithi16530038-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.892219503, 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.878598294, 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.621 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 814 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632168+0000", 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632177+0000", 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9500000000000007e-06 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632196+0000", 2024-09-16T04:42:55.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9296e-05 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632207+0000", 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.137e-05 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815302+0000", 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18309505200000001 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815453+0000", 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000151065 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342052+0000", 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52659867599999999 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342192+0000", 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013975200000000001 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.510764+0000", 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.168572213 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1es0 e100,97 3:78f34faf:::smithi16530038-13:f,90'68)", 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.3369296789999998, 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.63827547699999998, 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.626 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187458+0000", 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.12e-06 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187470+0000", 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2356e-05 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187502+0000", 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2135999999999999e-05 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187519+0000", 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6730000000000001e-05 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.823533+0000", 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.63601388800000003 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.825731+0000", 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002198247 2024-09-16T04:42:55.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1es0 e100,97 3:78f34faf:::smithi16530038-13:11,92'72)", 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.2263265539999999, 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.537458137, 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298060+0000", 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.607e-06 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298071+0000", 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.03e-05 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298103+0000", 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2308999999999998e-05 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298119+0000", 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5642999999999999e-05 2024-09-16T04:42:55.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.834381+0000", 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.53626253700000004 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.835517+0000", 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001135741 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.633 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:958 3.1es0 3:78f34faf:::smithi16530038-13:head [rollback 1d] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.92905973100000006, 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54156909600000003, 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.634 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 958 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595327+0000", 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.257e-06 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595333+0000", 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0959999999999997e-06 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595353+0000", 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9656999999999998e-05 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595375+0000", 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1982999999999999e-05 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.722826+0000", 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12745103099999999 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.723141+0000", 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031585100000000002 2024-09-16T04:42:55.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724986+0000", 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00184425 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.831289+0000", 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106303545 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.136810+0000", 2024-09-16T04:42:55.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30552102599999997 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.136895+0000", 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4400000000000005e-05 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:959 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.92898883099999996, 2024-09-16T04:42:55.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54178843700000001, 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 959 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595398+0000", 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.223e-06 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595404+0000", 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8239999999999996e-06 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595424+0000", 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0876000000000001e-05 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595433+0000", 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9099999999999994e-06 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.827455+0000", 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23202157200000001 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.827657+0000", 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000201709 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828666+0000", 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010089389999999999 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.831586+0000", 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029207149999999999 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.137124+0000", 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30553806500000003 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.137185+0000", 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0603999999999998e-05 2024-09-16T04:42:55.644 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.645 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.645 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.645 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:55.646 INFO:tasks.rados.rados.0.smithi165.stdout:392: done (4 left) 2024-09-16T04:42:55.646 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 32 current snap is 35 2024-09-16T04:42:55.646 INFO:tasks.rados.rados.0.smithi165.stdout:393: seq_num 142 ranges {1458176=442368,1900544=720896,2621440=540672,3162112=688128} 2024-09-16T04:42:55.647 INFO:tasks.rados.rados.0.smithi165.stdout:393: writing smithi16530038-32 from 1458176 to 1900544 tid 1 2024-09-16T04:42:55.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:42:55.648 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:55.642+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:55.658 INFO:tasks.rados.rados.0.smithi165.stdout:393: writing smithi16530038-32 from 1900544 to 2621440 tid 2 2024-09-16T04:42:55.665 INFO:tasks.rados.rados.0.smithi165.stdout:393: writing smithi16530038-32 from 2621440 to 3162112 tid 3 2024-09-16T04:42:55.675 INFO:tasks.rados.rados.0.smithi165.stdout:393: writing smithi16530038-32 from 3162112 to 3850240 tid 4 2024-09-16T04:42:55.675 INFO:tasks.rados.rados.0.smithi165.stdout:394: read oid 27 snap -1 2024-09-16T04:42:55.675 INFO:tasks.rados.rados.0.smithi165.stdout:394: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:55.675 INFO:tasks.rados.rados.0.smithi165.stdout:395: setattr oid 30 current snap is 35 2024-09-16T04:42:55.675 INFO:tasks.rados.rados.0.smithi165.stdout:388: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:42:55.686 INFO:tasks.rados.rados.0.smithi165.stdout:387: finishing rollback tid 0 to smithi16530038-13 2024-09-16T04:42:55.686 INFO:tasks.rados.rados.0.smithi165.stdout:387: finishing rollback tid 1 to smithi16530038-13 2024-09-16T04:42:55.686 INFO:tasks.rados.rados.0.smithi165.stdout:387: finishing rollback tid 2 to smithi16530038-13 2024-09-16T04:42:55.687 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 96 (ObjNum 86 snap 14 seq_num 86) dirty exists 2024-09-16T04:42:55.687 INFO:tasks.rados.rados.0.smithi165.stdout:389: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:42:55.703 INFO:tasks.rados.rados.0.smithi165.stdout:391: finishing rollback tid 0 to smithi16530038-46 2024-09-16T04:42:55.703 INFO:tasks.rados.rados.0.smithi165.stdout:391: finishing rollback tid 1 to smithi16530038-46 2024-09-16T04:42:55.704 INFO:tasks.rados.rados.0.smithi165.stdout:387: done (6 left) 2024-09-16T04:42:55.704 INFO:tasks.rados.rados.0.smithi165.stdout:388: done (5 left) 2024-09-16T04:42:55.704 INFO:tasks.rados.rados.0.smithi165.stdout:389: done (4 left) 2024-09-16T04:42:55.704 INFO:tasks.rados.rados.0.smithi165.stdout:append oid 37 current snap is 35 2024-09-16T04:42:55.704 INFO:tasks.rados.rados.0.smithi165.stdout:396: seq_num 143 ranges {2801664=458752,3260416=475136,3735552=655360,4390912=671744,5062656=147456} 2024-09-16T04:42:55.736 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16530038-37 from 2801664 to 3260416 tid 1 2024-09-16T04:42:55.740 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16530038-37 from 3260416 to 3735552 tid 2 2024-09-16T04:42:55.746 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16530038-37 from 3735552 to 4390912 tid 3 2024-09-16T04:42:55.747 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:55.742+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:55.752 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16530038-37 from 4390912 to 5062656 tid 4 2024-09-16T04:42:55.753 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16530038-37 from 5062656 to 5210112 tid 5 2024-09-16T04:42:55.753 INFO:tasks.rados.rados.0.smithi165.stdout:397: snap_create 2024-09-16T04:42:55.753 INFO:tasks.rados.rados.0.smithi165.stdout:391: finishing rollback tid 2 to smithi16530038-46 2024-09-16T04:42:55.753 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 54 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-16T04:42:55.829 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:55.829 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:55.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.829 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 105/67 ECSubWrite(tid=289, reqid=client.4728.0:974, at_version=105'110, trim_to=0'0, roll_forward_to=102'105))", 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.754960+0000", 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.073288136000000004, 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073347658999999996, 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.754960+0000", 2024-09-16T04:42:55.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.754960+0000", 2024-09-16T04:42:55.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.754961+0000", 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6400000000000001e-07 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755134+0000", 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017308300000000001 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755156+0000", 2024-09-16T04:42:55.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1803000000000001e-05 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755170+0000", 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3713000000000001e-05 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 105/67 ECSubWrite(tid=230, reqid=osd.4.0:230, at_version=105'83, trim_to=0'0, roll_forward_to=97'81))", 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.755715+0000", 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.072533735000000002, 2024-09-16T04:42:55.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.072756768999999999, 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755715+0000", 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755715+0000", 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:55.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755715+0000", 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0699999999999998e-07 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755742+0000", 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7226999999999999e-05 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:55.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755754+0000", 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2201999999999999e-05 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755765+0000", 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0875e-05 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 2 2024-09-16T04:42:55.837 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:55.846 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:42:55.847 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:55.843+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:55.948 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:55.943+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:55.972 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:55.972 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:55.972 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:55.972 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:55.972 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:55.986 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:42:56.059 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.15 2024-09-16T04:42:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:42:56.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:42:56.082 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:56.079+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:56.123 INFO:tasks.rados.rados.0.smithi165.stdout:394: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:42:56.137 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 1 to smithi16530038-37 2024-09-16T04:42:56.150 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:56.142+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:56.151 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 103 (ObjNum 33 snap 0 seq_num 1783474304) dirty exists 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 63.749014731999999, 2024-09-16T04:42:56.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20191694600000001, 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:42:56.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409856+0000", 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409917+0000", 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1023999999999997e-05 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.163 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409958+0000", 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0945000000000002e-05 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510088+0000", 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100130017 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510247+0000", 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015866599999999999 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510287+0000", 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0330999999999997e-05 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611689+0000", 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101401328 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611771+0000", 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2582000000000003e-05 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.165 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.687464898999998, 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246694400000001, 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471405+0000", 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471443+0000", 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7605000000000003e-05 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471463+0000", 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0452999999999999e-05 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.571565+0000", 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100102105 2024-09-16T04:42:56.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671828+0000", 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026227 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671853+0000", 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5826e-05 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673802+0000", 2024-09-16T04:42:56.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019481730000000001 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673871+0000", 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9624999999999998e-05 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 69/67 ECSubWrite(tid=149, reqid=client.4728.0:576, at_version=69'48, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 36.647924037000003, 2024-09-16T04:42:56.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30202165600000003, 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510947+0000", 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.95e-06 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510983+0000", 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6198999999999997e-05 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.511014+0000", 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1112000000000003e-05 2024-09-16T04:42:56.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.611120+0000", 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100105729 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811521+0000", 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040100599999999 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811553+0000", 2024-09-16T04:42:56.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1958999999999997e-05 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812884+0000", 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013312700000000001 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812967+0000", 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2430999999999997e-05 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 70/67 ECSubWrite(tid=155, reqid=osd.4.0:153, at_version=0'0, trim_to=0'0, roll_forward_to=70'52))", 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 35.593003537000001, 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137054300000001, 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:56.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565867+0000", 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.84e-06 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565893+0000", 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5715999999999999e-05 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565933+0000", 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0358999999999998e-05 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666040+0000", 2024-09-16T04:42:56.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010693499999999 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666191+0000", 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015013299999999999 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666251+0000", 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0615000000000002e-05 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767150+0000", 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100898584 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767236+0000", 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6360999999999995e-05 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=59, reqid=osd.8.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:42:56.179 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.655913718000001, 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26917078799999999, 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502956+0000", 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3e-07 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502980+0000", 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3802e-05 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.503017+0000", 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7082999999999997e-05 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603116+0000", 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098623 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603238+0000", 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012259700000000001 2024-09-16T04:42:56.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603255+0000", 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6762999999999999e-05 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772084+0000", 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16882903799999999 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772126+0000", 2024-09-16T04:42:56.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2151999999999999e-05 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=97, reqid=client.4728.0:613, at_version=74'68, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.591924898999999, 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20513274000000001, 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566945+0000", 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.03e-07 2024-09-16T04:42:56.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567199+0000", 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025406099999999998 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567228+0000", 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9589000000000001e-05 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567244+0000", 2024-09-16T04:42:56.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6027000000000001e-05 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567424+0000", 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000180006 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567456+0000", 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1251999999999998e-05 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772018+0000", 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20456203000000001 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772077+0000", 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9372e-05 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=98, reqid=client.4728.0:614, at_version=74'69, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:42:56.188 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.59158438, 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.205026973, 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:42:56.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567286+0000", 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567509+0000", 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022361500000000001 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567536+0000", 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7063000000000002e-05 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567555+0000", 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669312+0000", 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101756135 2024-09-16T04:42:56.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669353+0000", 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1069000000000003e-05 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772248+0000", 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102895556 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772312+0000", 2024-09-16T04:42:56.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3559000000000006e-05 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=74, reqid=client.4728.0:707, at_version=0'0, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 25.786595263999999, 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202706724, 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372276+0000", 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.782e-06 2024-09-16T04:42:56.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372297+0000", 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0907000000000002e-05 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372320+0000", 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3759000000000001e-05 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372333+0000", 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2719e-05 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472559+0000", 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10022637399999999 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472570+0000", 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0351e-05 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574932+0000", 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10236213700000001 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574981+0000", 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8695e-05 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.197 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=75, reqid=client.4728.0:708, at_version=79'59, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 25.784176862999999, 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20032509600000001, 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374694+0000", 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374738+0000", 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3016000000000003e-05 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374782+0000", 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4709999999999997e-05 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374813+0000", 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0700000000000001e-05 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473059+0000", 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098245669999999993 2024-09-16T04:42:56.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473071+0000", 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.24e-05 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574989+0000", 2024-09-16T04:42:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10191757 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575017+0000", 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8773e-05 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 80/67 ECSubWrite(tid=114, reqid=osd.2.0:114, at_version=80'75, trim_to=0'0, roll_forward_to=79'73))", 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 25.765025978000001, 2024-09-16T04:42:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18121299299999999, 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393843+0000", 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393844+0000", 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5000000000000003e-07 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393900+0000", 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6436e-05 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393925+0000", 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4703999999999999e-05 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.393948+0000", 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3173e-05 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473572+0000", 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079623997000000002 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473583+0000", 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1327e-05 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575021+0000", 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101437639 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575056+0000", 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5166999999999997e-05 2024-09-16T04:42:56.206 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=179, reqid=client.4728.0:716, at_version=81'60, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 24.618565745000001, 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20327186, 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540304+0000", 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.73e-07 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540825+0000", 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00052150999999999999 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540843+0000", 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8321e-05 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540858+0000", 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.452e-05 2024-09-16T04:42:56.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.741263+0000", 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040504000000001 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.741281+0000", 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7946999999999999e-05 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.743496+0000", 2024-09-16T04:42:56.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0022152199999999999 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.743575+0000", 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9128999999999994e-05 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=180, reqid=client.4728.0:717, at_version=81'61, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 24.612185251, 2024-09-16T04:42:56.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29802481800000002, 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:42:56.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546685+0000", 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4100000000000001e-06 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547131+0000", 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00044599499999999997 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547146+0000", 2024-09-16T04:42:56.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5145e-05 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547155+0000", 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8710000000000003e-06 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.742419+0000", 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.195263674 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.742437+0000", 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.84e-05 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.844629+0000", 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102192143 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.844709+0000", 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9179999999999997e-05 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.215 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=181, reqid=client.4728.0:718, at_version=81'62, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.650122+0000", 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 24.508747429, 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19530381299999999, 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650122+0000", 2024-09-16T04:42:56.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650122+0000", 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650124+0000", 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.514e-06 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650371+0000", 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024698100000000001 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650415+0000", 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3943000000000003e-05 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.650442+0000", 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.656e-05 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.843720+0000", 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19327802499999999 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.843736+0000", 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6691e-05 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.845381+0000", 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016446239999999999 2024-09-16T04:42:56.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.845425+0000", 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4474999999999997e-05 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 90/69 ECSubWrite(tid=84, reqid=osd.8.0:84, at_version=90'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.566569521, 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19829587900000001, 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.220 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592301+0000", 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5430000000000001e-06 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592337+0000", 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6075999999999998e-05 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592365+0000", 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7863000000000001e-05 2024-09-16T04:42:56.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592380+0000", 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.504e-05 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.788683+0000", 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19630244899999999 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.788698+0000", 2024-09-16T04:42:56.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5113000000000001e-05 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.790504+0000", 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018063 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.790595+0000", 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1495000000000004e-05 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s6 92/55 ECSubWrite(tid=97, reqid=osd.5.0:97, at_version=92'31, trim_to=0'0, roll_forward_to=90'29))", 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.555015906, 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.203223405, 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603854+0000", 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6700000000000003e-07 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603919+0000", 2024-09-16T04:42:56.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5079000000000005e-05 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603953+0000", 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4215999999999999e-05 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603970+0000", 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6626999999999999e-05 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.804328+0000", 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20035859 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.804344+0000", 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5252999999999999e-05 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.807006+0000", 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026628329999999999 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.807077+0000", 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0140000000000003e-05 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPush(3.1as5 101/97 [PushOp(3:5f4281cc:::smithi16530038-45:1a, version: 90'54, data_included: [262144~217088], data_size: 217088, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5f4281cc:::smithi16530038-45:1a@90'54, size: 1916928, copy_subset: [], clone_subset: {}, snapset: 1f=[]:{1a: [1a, 19, 18], 1f: [1f, 1e, 1d, 1b]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1916928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.592787+0000", 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 3.566081879, 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19797579700000001, 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.229 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.592787+0000", 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.592787+0000", 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.592788+0000", 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593214+0000", 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000425597 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593272+0000", 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7960000000000001e-05 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593325+0000", 2024-09-16T04:42:56.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3965999999999998e-05 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.789316+0000", 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19599050200000001 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.790763+0000", 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001447039 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.232 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es5 104/97 ECSubWrite(tid=198, reqid=client.4728.0:957, at_version=104'94, trim_to=0'0, roll_forward_to=99'92))", 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.719871+0000", 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.43899858, 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28552035100000001, 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719871+0000", 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719871+0000", 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719872+0000", 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.813e-06 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719919+0000", 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6553e-05 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719953+0000", 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3912999999999999e-05 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719969+0000", 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.66e-05 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.806281+0000", 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.08631192 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.806307+0000", 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5205999999999999e-05 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005331+0000", 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.199024957 2024-09-16T04:42:56.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005391+0000", 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9388999999999997e-05 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es5 104/97 ECSubWrite(tid=199, reqid=client.4728.0:958, at_version=104'95, trim_to=0'0, roll_forward_to=99'92))", 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.724906+0000", 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.4339627749999999, 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28052565600000001, 2024-09-16T04:42:56.237 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724906+0000", 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724906+0000", 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724908+0000", 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7069999999999999e-06 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724962+0000", 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3808999999999999e-05 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724998+0000", 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5819999999999999e-05 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.725022+0000", 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4593e-05 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.808230+0000", 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.083207507 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.808257+0000", 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6812999999999999e-05 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005399+0000", 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19714209399999999 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005432+0000", 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3312999999999998e-05 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.241 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es5 104/97 ECSubWrite(tid=200, reqid=client.4728.0:959, at_version=104'96, trim_to=0'0, roll_forward_to=99'92))", 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.828630+0000", 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.3302395200000001, 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.176831971, 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828630+0000", 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828630+0000", 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828631+0000", 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.59e-06 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828675+0000", 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4039999999999998e-05 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828718+0000", 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2979999999999998e-05 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828746+0000", 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8269e-05 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828913+0000", 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016658499999999999 2024-09-16T04:42:56.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828945+0000", 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1874e-05 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005435+0000", 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.176490226 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005462+0000", 2024-09-16T04:42:56.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6407e-05 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 105/67 ECSubWrite(tid=289, reqid=client.4728.0:974, at_version=105'110, trim_to=0'0, roll_forward_to=102'105))", 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.754960+0000", 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.40390887599999997, 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19787004499999999, 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.754960+0000", 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.754960+0000", 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.754961+0000", 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6400000000000001e-07 2024-09-16T04:42:56.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755134+0000", 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017308300000000001 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755156+0000", 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1803000000000001e-05 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.755170+0000", 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3713000000000001e-05 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.950536+0000", 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19536667399999999 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.950548+0000", 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1449e-05 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.952740+0000", 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0021925970000000001 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.952830+0000", 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9962000000000004e-05 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.250 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:56.252 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:42:56.253 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:56.247+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:56.253 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 2 to smithi16530038-37 2024-09-16T04:42:56.257 INFO:tasks.rados.rados.0.smithi165.stdout:393: finishing write tid 1 to smithi16530038-32 2024-09-16T04:42:56.353 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:56.346+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:56.354 INFO:tasks.rados.rados.0.smithi165.stdout:393: finishing write tid 2 to smithi16530038-32 2024-09-16T04:42:56.355 INFO:tasks.rados.rados.0.smithi165.stdout:393: finishing write tid 3 to smithi16530038-32 2024-09-16T04:42:56.356 INFO:tasks.rados.rados.0.smithi165.stdout:393: finishing write tid 4 to smithi16530038-32 2024-09-16T04:42:56.357 INFO:tasks.rados.rados.0.smithi165.stdout:393: finishing write tid 5 to smithi16530038-32 2024-09-16T04:42:56.390 INFO:teuthology.orchestra.run.smithi165.stderr:Exporting 3.2s4 info 3.2s4( v 59'15 (0'0,59'15] local-lis/les=67/68 n=1 ec=32/32 lis/c=67/67 les/c/f=68/68/0 sis=67) 2024-09-16T04:42:56.397 INFO:teuthology.orchestra.run.smithi165.stderr:Export successful 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPush(3.5s0 105/102 [PushOp(3:a54c6f35:::smithi16530038-38:head, version: 92'80, data_included: [262144~253952], data_size: 253952, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:a54c6f35:::smithi16530038-38:head@92'80, size: 2064384, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2064384, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:56.343775+0000", 2024-09-16T04:42:56.423 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.078873657, 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.078959420000000002, 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "no flag points reached", 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.343775+0000", 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.424 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.343775+0000", 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.343775+0000", 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.343775+0000", 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.343878+0000", 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010254 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpRead(3.cs3 105/67 ECSubRead(tid=294, to_read={3:372a9655:::smithi16530038-32:head=0,4096,0}, subchunks={3:372a9655:::smithi16530038-32:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:56.357841+0000", 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.064808131000000005, 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.064979254, 2024-09-16T04:42:56.426 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.357841+0000", 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.357841+0000", 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.427 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.357842+0000", 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1599999999999999e-06 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.357854+0000", 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2330000000000001e-05 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.357876+0000", 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1872999999999999e-05 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.357889+0000", 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2373e-05 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpReadReply(3.11s0 105/102 ECSubReadReply(tid=52, attrs_read=0))", 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:56.366774+0000", 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.055874864000000003, 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.056120449000000003, 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.429 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.366774+0000", 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.366774+0000", 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.430 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.366775+0000", 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2240000000000001e-06 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.366856+0000", 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0461000000000001e-05 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.366873+0000", 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6663999999999998e-05 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.366891+0000", 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8559999999999998e-05 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 3 2024-09-16T04:42:56.432 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:56.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:42:56.449 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 3 to smithi16530038-37 2024-09-16T04:42:56.449 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 4 to smithi16530038-37 2024-09-16T04:42:56.449 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 5 to smithi16530038-37 2024-09-16T04:42:56.450 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 6 to smithi16530038-37 2024-09-16T04:42:56.482 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:56.479+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:56.556 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:56.550+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:393: finishing write tid 6 to smithi16530038-32 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:393: oid 32 updating version 0 to 111 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:393: oid 32 version 111 is already newer than 108 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:393: oid 32 version 111 is already newer than 109 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:393: oid 32 version 111 is already newer than 110 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:393: oid 32 version 111 is already newer than 107 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 111 (ObjNum 142 snap 35 seq_num 142) dirty exists 2024-09-16T04:42:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:393: left oid 32 (ObjNum 142 snap 35 seq_num 142) 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:56.562 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:56.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:391: done (5 left) 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:393: done (4 left) 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:394: done (3 left) 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:395: done (2 left) 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:397: done (1 left) 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:398: read oid 50 snap 27 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:398: expect (ObjNum 120 snap 24 seq_num 120) 2024-09-16T04:42:56.614 INFO:tasks.rados.rados.0.smithi165.stdout:399: snap_remove snap 33 2024-09-16T04:42:56.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:42:56.658 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:56.650+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:56.745 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.487987066999999, 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.396784737, 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.46e-07 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253567+0000", 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00085024700000000005 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253594+0000", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6529e-05 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253616+0000", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2373000000000001e-05 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448095+0000", 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19447869800000001 2024-09-16T04:42:56.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448103+0000", 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9379999999999992e-06 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649474+0000", 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137053299999999 2024-09-16T04:42:56.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649502+0000", 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7973000000000001e-05 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:56.750 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.482288912999998, 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.391121679, 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:42:56.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258417+0000", 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.722e-06 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258658+0000", 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024151700000000001 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258682+0000", 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3526000000000001e-05 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258695+0000", 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3225e-05 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548388+0000", 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28969289100000001 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548393+0000", 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2970000000000003e-06 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649523+0000", 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101130161 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649537+0000", 2024-09-16T04:42:56.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3339999999999999e-05 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.479822273000003, 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38866693400000002, 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260883+0000", 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0750000000000001e-06 2024-09-16T04:42:56.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260979+0000", 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6552999999999995e-05 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260998+0000", 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.261010+0000", 2024-09-16T04:42:56.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1817999999999999e-05 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548799+0000", 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28778851599999999 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548839+0000", 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0371999999999999e-05 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649539+0000", 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100699682 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649549+0000", 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8050000000000001e-06 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:42:56.759 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.478183975, 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38775422500000001, 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262521+0000", 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.009e-06 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262542+0000", 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0981000000000001e-05 2024-09-16T04:42:56.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262560+0000", 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8224e-05 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262572+0000", 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2155e-05 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649287+0000", 2024-09-16T04:42:56.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38671449499999999 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649295+0000", 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8329999999999994e-06 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650254+0000", 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00095943799999999996 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650274+0000", 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0089999999999999e-05 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 36.317663609, 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.32087775200000002, 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.764 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423042+0000", 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.77e-06 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423051+0000", 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8100000000000004e-06 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423081+0000", 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0626000000000001e-05 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523175+0000", 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100093245 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742603+0000", 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21942863300000001 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.743918+0000", 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001314668 2024-09-16T04:42:56.767 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:937 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.580620+0000", 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.1600834579999999, 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.132113506, 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.768 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 937 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580620+0000", 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580620+0000", 2024-09-16T04:42:56.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580624+0000", 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4800000000000001e-06 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580657+0000", 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3176000000000002e-05 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580693+0000", 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6028999999999997e-05 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580711+0000", 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8377e-05 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580874+0000", 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016261199999999999 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580891+0000", 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7263999999999999e-05 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.799454+0000", 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21856298099999999 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.799462+0000", 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6170000000000001e-06 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706333+0000", 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90687079400000004 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706338+0000", 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5600000000000001e-06 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.912371+0000", 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.206032937 2024-09-16T04:42:56.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for degraded object", 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.913944+0000", 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015726609999999999 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.395897+0000", 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.481953138 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.396116+0000", 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00021892799999999999 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.397583+0000", 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014675319999999999 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.502978+0000", 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105394321 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.712199+0000", 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20922146 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.712734+0000", 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00053463899999999995 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.776 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:938 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [rollback 1e] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.580748+0000", 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.1599556890000002, 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.232328887, 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 938 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580748+0000", 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580748+0000", 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580750+0000", 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.55e-06 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580757+0000", 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7300000000000005e-06 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580780+0000", 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2872999999999999e-05 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580796+0000", 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5722999999999998e-05 2024-09-16T04:42:56.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.581076+0000", 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027972400000000002 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.581101+0000", 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4749999999999999e-05 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800699+0000", 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21959847099999999 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800704+0000", 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3639999999999998e-06 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706434+0000", 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90572917799999997 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706439+0000", 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9840000000000001e-06 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914075+0000", 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2076361680000001 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for degraded object", 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914228+0000", 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000153549 2024-09-16T04:42:56.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399055+0000", 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48482694199999998 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399427+0000", 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000372011 2024-09-16T04:42:56.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.401491+0000", 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020635300000000001 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.503165+0000", 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10167448699999999 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:56.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.812292+0000", 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30912664899999998 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813077+0000", 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00078520399999999996 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:939 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:42:56.785 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 5.1598863770000003, 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2334956030000002, 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 939 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:42:56.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580819+0000", 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.297e-06 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580825+0000", 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9499999999999998e-06 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580864+0000", 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9606e-05 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.681043+0000", 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10017833499999999 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800804+0000", 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11976181399999999 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800808+0000", 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6430000000000001e-06 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.008117+0000", 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.207308388 2024-09-16T04:42:56.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.008122+0000", 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5799999999999999e-06 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914306+0000", 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.906183721 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for degraded object", 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914561+0000", 2024-09-16T04:42:56.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025529200000000001 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.402786+0000", 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48822523899999998 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.403012+0000", 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000225448 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.404166+0000", 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011544509999999999 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.503170+0000", 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099003718000000004 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813807+0000", 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31063750800000001 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.814313+0000", 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00050561300000000003 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:943 3.5s0 3:a54da884:::smithi16530038-14:head [copy-from ver 0] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.584047+0000", 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.156656806, 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3330712170000001, 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.793 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 943 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584047+0000", 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584047+0000", 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584051+0000", 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7359999999999999e-06 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584066+0000", 2024-09-16T04:42:56.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.518e-05 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584103+0000", 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.748e-05 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584126+0000", 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2857000000000002e-05 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593972+0000", 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0098458580000000007 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593979+0000", 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5769999999999999e-06 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.916665+0000", 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.322686647 2024-09-16T04:42:56.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.919068+0000", 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024021889999999999 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.026141+0000", 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10707359299999999 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.027046+0000", 2024-09-16T04:42:56.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00090457999999999999 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.111730+0000", 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.084683865999999997 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.112416+0000", 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00068637699999999995 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.799 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.917118+0000", 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80470227699999997 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:944 3.5s0 3:a54da884:::smithi16530038-14:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.584174+0000", 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.1565299739999997, 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3352599700000001, 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:56.800 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 944 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584174+0000", 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584174+0000", 2024-09-16T04:42:56.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584175+0000", 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.587e-06 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584183+0000", 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2429999999999999e-06 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584206+0000", 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3637000000000001e-05 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584224+0000", 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.738e-05 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.595503+0000", 2024-09-16T04:42:56.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011279697999999999 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.595510+0000", 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1929999999999998e-06 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.924019+0000", 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3285094630000001 2024-09-16T04:42:56.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.924445+0000", 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00042564700000000003 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.027556+0000", 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103110966 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.028442+0000", 2024-09-16T04:42:56.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00088608700000000003 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.114030+0000", 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.085588417999999999 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.114447+0000", 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00041656400000000002 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.917592+0000", 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80314557499999994 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.918393+0000", 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000800707 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.919434+0000", 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010408049999999999 2024-09-16T04:42:56.807 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=38, last_complete=38'14, committed=1, applied=1))", 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.399580+0000", 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 2.3411234809999999, 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31204127700000001, 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399580+0000", 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399580+0000", 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399582+0000", 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4869999999999999e-06 2024-09-16T04:42:56.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399590+0000", 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2169999999999994e-06 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399613+0000", 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2442999999999998e-05 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.499725+0000", 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100112408 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.711569+0000", 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.211843632 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.711622+0000", 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3090000000000002e-05 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.811 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=40, last_complete=103'50, committed=1, applied=1))", 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.500039+0000", 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 2.240664615, 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31329262800000002, 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.812 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500039+0000", 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500039+0000", 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500040+0000", 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1699999999999997e-07 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500044+0000", 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9530000000000001e-06 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500055+0000", 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0672999999999999e-05 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500062+0000", 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9439999999999999e-06 2024-09-16T04:42:56.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813246+0000", 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31318400800000001 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813332+0000", 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6333000000000002e-05 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=40, last_complete=38'14, committed=1, applied=1))", 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.504051+0000", 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 2.2366527220000001, 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30950637600000003, 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504051+0000", 2024-09-16T04:42:56.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504051+0000", 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504053+0000", 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.39e-06 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504060+0000", 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5100000000000001e-06 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504086+0000", 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6336999999999999e-05 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504101+0000", 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4382999999999999e-05 2024-09-16T04:42:56.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813516+0000", 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30941474699999999 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813558+0000", 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2008999999999997e-05 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=39, last_complete=103'49, committed=1, applied=1))", 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.504553+0000", 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 2.2361506919999998, 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30856470699999999, 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504553+0000", 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504553+0000", 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504554+0000", 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7000000000000005e-07 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504558+0000", 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7759999999999999e-06 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504571+0000", 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3773e-05 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504580+0000", 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2640000000000008e-06 2024-09-16T04:42:56.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.811967+0000", 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.307387615 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813118+0000", 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001150709 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=40, last_complete=103'50, committed=1, applied=1))", 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.506035+0000", 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 2.2346690539999998, 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30937354099999997, 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506035+0000", 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506035+0000", 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506036+0000", 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2270000000000001e-06 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506043+0000", 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3170000000000004e-06 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506066+0000", 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2195999999999999e-05 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506078+0000", 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2807e-05 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813628+0000", 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30755011399999999 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.815408+0000", 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00177988 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:967 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.114620+0000", 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.626084299, 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41900611999999998, 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 967 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.828 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114620+0000", 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114620+0000", 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114622+0000", 2024-09-16T04:42:56.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.187e-06 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114631+0000", 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3530000000000001e-06 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114656+0000", 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.476e-05 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114676+0000", 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9706000000000001e-05 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.121725+0000", 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0070496040000000001 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.122126+0000", 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040128000000000001 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.124630+0000", 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002503365 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.129806+0000", 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051757340000000004 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.533049+0000", 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40324364499999998 2024-09-16T04:42:56.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.533626+0000", 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00057648599999999997 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:968 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [rollback 23] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.114703+0000", 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.6260003629999999, 2024-09-16T04:42:56.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.420739427, 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 968 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114703+0000", 2024-09-16T04:42:56.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114703+0000", 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114705+0000", 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3069999999999999e-06 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114711+0000", 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4269999999999996e-06 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114732+0000", 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0780000000000001e-05 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114748+0000", 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5613000000000001e-05 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.126283+0000", 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011535665000000001 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.126717+0000", 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000433554 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.128150+0000", 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001433372 2024-09-16T04:42:56.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.130216+0000", 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020659279999999999 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.534414+0000", 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40419767499999998 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.535443+0000", 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010291059999999999 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:969 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.114770+0000", 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.625933743, 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.59494319200000001, 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:56.839 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 969 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114770+0000", 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114770+0000", 2024-09-16T04:42:56.840 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114771+0000", 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2300000000000001e-06 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114778+0000", 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1800000000000001e-06 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.841 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114797+0000", 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9162999999999999e-05 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.214897+0000", 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009994799999999 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.218214+0000", 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0033174680000000001 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.218451+0000", 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023720499999999999 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.219546+0000", 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010942879999999999 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.220601+0000", 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010550430000000001 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.709185+0000", 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.488584507 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.709713+0000", 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00052815999999999996 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.11s0 e105,102 3:882e132b:::smithi16530038-22:head,92'42)", 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.714221+0000", 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.0264826970000001, 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39956194499999997, 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714221+0000", 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714221+0000", 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714222+0000", 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.032e-06 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714230+0000", 2024-09-16T04:42:56.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3560000000000004e-06 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714245+0000", 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5398999999999999e-05 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714253+0000", 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5259999999999993e-06 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.847 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.112728+0000", 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39847491299999999 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.113783+0000", 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001054719 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.21s4 105/97 ECSubWrite(tid=229, reqid=osd.4.0:229, at_version=105'49, trim_to=0'0, roll_forward_to=99'47))", 2024-09-16T04:42:56.848 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.008101717, 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38255834700000002, 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:56.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.58e-07 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.850 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732631+0000", 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8413000000000001e-05 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732648+0000", 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6932999999999999e-05 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732663+0000", 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4901e-05 2024-09-16T04:42:56.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.912643+0000", 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.17997981800000001 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.912656+0000", 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3485e-05 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.115112+0000", 2024-09-16T04:42:56.852 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20245636 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.115160+0000", 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8078999999999998e-05 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:56.853 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:56.855 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:42:56.855 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:56.850+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:56.956 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:56.950+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:56.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-15 --journal-path /var/lib/ceph/osd/ceph-15/journal --force --op remove --pgid 3.2s4 2024-09-16T04:42:56.986 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:56.986 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:56.986 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:42:56.986 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:57.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:42:57.057 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:57.050+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:57.121 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:57.121 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:42:57.121 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:42:57.121 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:42:57.122 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:57.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:42:57.182 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:57.179+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:57.259 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:42:57.254+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:42:57.262 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:158 3.9s0 3:9de467d5:::smithi16530038-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 70.773976343000001, 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51808159200000004, 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.263 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 158 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:42:57.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484332+0000", 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484339+0000", 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8630000000000002e-06 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484355+0000", 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6473e-05 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484364+0000", 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8950000000000004e-06 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.586917+0000", 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102553035 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.587177+0000", 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026005499999999998 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.588403+0000", 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001225152 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589911+0000", 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015086260000000001 2024-09-16T04:42:57.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002165+0000", 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41225355800000002 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002413+0000", 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024783200000000002 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.9s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 70.773907534000003, 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.62506688499999996, 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.269 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484401+0000", 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484405+0000", 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.827e-06 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484416+0000", 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1609e-05 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484425+0000", 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790928+0000", 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30650327300000002 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.791067+0000", 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013860200000000001 2024-09-16T04:42:57.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003346+0000", 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212279577 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003611+0000", 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026456899999999998 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109440+0000", 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105828857 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109467+0000", 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6900999999999999e-05 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.19s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:57.274 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 70.150954927000001, 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.976424197, 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107354+0000", 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9829999999999999e-06 2024-09-16T04:42:57.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.277 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107362+0000", 2024-09-16T04:42:57.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6129999999999998e-06 2024-09-16T04:42:57.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107388+0000", 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5675000000000001e-05 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207482+0000", 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009412299999999 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207644+0000", 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000161741 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207649+0000", 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0359999999999997e-06 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082558+0000", 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8749096240000001 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082784+0000", 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022614900000000001 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.083777+0000", 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099225300000000002 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:178 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 69.180193501000005, 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1069151189999999, 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 178 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078116+0000", 2024-09-16T04:42:57.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078127+0000", 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.062e-05 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078151+0000", 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4304999999999999e-05 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078175+0000", 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4006e-05 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776496+0000", 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69832100500000005 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776503+0000", 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.533e-06 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.284 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185014+0000", 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40851148700000001 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185029+0000", 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4780000000000001e-05 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:179 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.285 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 69.180098138999995, 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1068670979999999, 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 179 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078211+0000", 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.44e-06 2024-09-16T04:42:57.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078220+0000", 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078244+0000", 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3867999999999999e-05 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078260+0000", 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6203000000000001e-05 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780436+0000", 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70217526500000005 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780441+0000", 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4620000000000004e-06 2024-09-16T04:42:57.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185067+0000", 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40462590799999998 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185076+0000", 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3219999999999999e-06 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:180 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 69.180025083000004, 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.106828843, 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 180 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078284+0000", 2024-09-16T04:42:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078291+0000", 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.657e-06 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078310+0000", 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078325+0000", 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4772e-05 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981190+0000", 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90286589500000003 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981194+0000", 2024-09-16T04:42:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0790000000000002e-06 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185100+0000", 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20390586399999999 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185111+0000", 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0794e-05 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:185 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 69.159709104000001, 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98955797199999995, 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098600+0000", 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098611+0000", 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0423e-05 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098642+0000", 2024-09-16T04:42:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1532000000000002e-05 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098658+0000", 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5996000000000001e-05 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683726+0000", 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58506727400000003 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683731+0000", 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088142+0000", 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40441113000000001 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088156+0000", 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4148000000000001e-05 2024-09-16T04:42:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:186 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 69.159621048999995, 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.089963279, 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.301 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 186 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098687+0000", 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.063e-06 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098693+0000", 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6230000000000002e-06 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098705+0000", 2024-09-16T04:42:57.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2019e-05 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198804+0000", 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098513 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683799+0000", 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48499579100000001 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.304 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683801+0000", 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0889999999999998e-06 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188641+0000", 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50483997300000005 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188650+0000", 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.208e-06 2024-09-16T04:42:57.305 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:187 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 69.059471646000006, 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98988477799999997, 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.306 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 187 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198838+0000", 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.86e-06 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198846+0000", 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0059999999999992e-06 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198864+0000", 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8468999999999999e-05 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198876+0000", 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1443e-05 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783957+0000", 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58508126900000001 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783962+0000", 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7230000000000001e-06 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188710+0000", 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40474714899999997 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188721+0000", 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0859000000000001e-05 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.310 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:583 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 37.923985690000002, 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2114807170000002, 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.311 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 583 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.312 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334325+0000", 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6969999999999999e-06 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334340+0000", 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4508999999999999e-05 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334388+0000", 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8016000000000002e-05 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334413+0000", 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5284999999999999e-05 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828480+0000", 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406639099999999 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828485+0000", 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3430000000000003e-06 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.769267+0000", 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.94078222199999995 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.771858+0000", 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025913479999999998 2024-09-16T04:42:57.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174292+0000", 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40243307299999997 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174565+0000", 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027297299999999998 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.176259+0000", 2024-09-16T04:42:57.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001694161 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281556+0000", 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10529764799999999 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545252+0000", 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.263695658 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545802+0000", 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00055039300000000004 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:584 3.1as0 3:5f4281cc:::smithi16530038-45:head [rollback 11] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 37.923851227, 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2123693160000002, 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.318 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 584 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334458+0000", 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.663e-06 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334465+0000", 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2239999999999998e-06 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334485+0000", 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9925999999999999e-05 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334500+0000", 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5449000000000001e-05 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828570+0000", 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49407003700000002 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828576+0000", 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.93e-06 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872096+0000", 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.043520019 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872408+0000", 2024-09-16T04:42:57.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031130200000000001 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177318+0000", 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30491066900000002 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177579+0000", 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026102699999999999 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.178857+0000", 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001277395 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281984+0000", 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10312769099999999 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546082+0000", 2024-09-16T04:42:57.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26409753200000002 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546825+0000", 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00074345200000000002 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:585 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:57.325 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 37.923787355000002, 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3685253789999998, 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 585 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:57.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334521+0000", 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4169999999999999e-06 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334529+0000", 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8560000000000007e-06 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334547+0000", 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7813e-05 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334561+0000", 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3803e-05 2024-09-16T04:42:57.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828622+0000", 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406153000000003 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828624+0000", 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.277e-06 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873077+0000", 2024-09-16T04:42:57.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.044453106 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873349+0000", 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027208400000000002 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279758+0000", 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40640956900000003 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.330 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279965+0000", 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000206028 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281051+0000", 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010864399999999999 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.282237+0000", 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011862839999999999 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.701843+0000", 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41960623800000002 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.703045+0000", 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001201934 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.332 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:635 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~524288] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 31.084927746000002, 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71243645600000005, 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 635 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.333 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173382+0000", 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174349+0000", 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00096718499999999996 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174408+0000", 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9036000000000002e-05 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174432+0000", 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4176e-05 2024-09-16T04:42:57.335 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579526+0000", 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40509398299999999 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579719+0000", 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019240699999999999 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.581395+0000", 2024-09-16T04:42:57.336 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016763769999999999 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.586083+0000", 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0046879950000000004 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885565+0000", 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29948223000000002 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.337 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885816+0000", 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025068399999999999 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:799 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~524288] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.765340695999999, 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.61375249600000004, 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.338 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 799 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.339 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492970+0000", 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.027e-06 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.493969+0000", 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099961899999999998 2024-09-16T04:42:57.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494004+0000", 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5166000000000002e-05 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494056+0000", 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1807000000000001e-05 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494238+0000", 2024-09-16T04:42:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018187800000000001 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494252+0000", 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.349e-05 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.593780+0000", 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099528025000000006 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.594015+0000", 2024-09-16T04:42:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023556099999999999 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.596480+0000", 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024650140000000002 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803670+0000", 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.207189436 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.343 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.106478+0000", 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280800499999999 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.106719+0000", 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000241468 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.344 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:800 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~573440] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.760873115999999, 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70919317999999998, 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 800 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.345 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497437+0000", 2024-09-16T04:42:57.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.796e-06 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498329+0000", 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000892124 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498363+0000", 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3908999999999997e-05 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.347 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498379+0000", 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6124e-05 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498519+0000", 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000140245 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:57.348 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498535+0000", 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.579e-05 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.599170+0000", 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100634526 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.599348+0000", 2024-09-16T04:42:57.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017774199999999999 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.600830+0000", 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001482178 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803675+0000", 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20284549599999999 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206492+0000", 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40281652400000001 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206627+0000", 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013572599999999999 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.351 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:801 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~688128] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.755671069000002, 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70428581800000001, 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 801 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502639+0000", 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:42:57.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503645+0000", 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001006002 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503676+0000", 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1526999999999998e-05 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503692+0000", 2024-09-16T04:42:57.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5962999999999999e-05 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503829+0000", 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000137229 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503845+0000", 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5270000000000001e-05 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.355 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.602063+0000", 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098218277000000007 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.602224+0000", 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000160996 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.603888+0000", 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016640330000000001 2024-09-16T04:42:57.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803679+0000", 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19979068899999999 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206810+0000", 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40313103500000003 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206922+0000", 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011254 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:802 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~294912] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.753540631, 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70333736700000005, 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.358 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 802 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.359 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504768+0000", 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.48e-06 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505226+0000", 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00045742599999999998 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.360 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505255+0000", 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9810000000000001e-05 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505276+0000", 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0639999999999999e-05 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505379+0000", 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000103169 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505389+0000", 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8700000000000004e-06 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.798131+0000", 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29274190900000002 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.798322+0000", 2024-09-16T04:42:57.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000190912 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.799632+0000", 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001310173 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803863+0000", 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0042312249999999999 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.207875+0000", 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40401177799999999 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.208104+0000", 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000228975 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:803 3.1as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (57) in=7b] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:42:57.364 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.752996219, 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.010178128, 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 803 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.365 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505313+0000", 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.533e-06 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.366 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505320+0000", 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6930000000000007e-06 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505340+0000", 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.003e-05 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605448+0000", 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010718 2024-09-16T04:42:57.367 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.210631+0000", 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60518353999999996 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.210911+0000", 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027993799999999998 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.212558+0000", 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016472839999999999 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.315431+0000", 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102872511 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.515321+0000", 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19989022000000001 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.515489+0000", 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016819900000000001 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:804 3.1as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.652824663000001, 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3152593989999999, 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.370 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 804 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605485+0000", 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9230000000000001e-06 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605493+0000", 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4069999999999993e-06 2024-09-16T04:42:57.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605516+0000", 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2506000000000001e-05 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605528+0000", 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2734e-05 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.373 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.313900+0000", 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70837135200000001 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.314072+0000", 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017190200000000001 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.516098+0000", 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20202645 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.516435+0000", 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000336535 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.920742+0000", 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40430758999999999 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:952 3.19s0 3:9e9173cd:::smithi16530038-4:head [create,setxattr kVnkrXsmOsSXXn[NoPlghhiQY]Z[N[YpkZhhkWli (1611) in=40b,setxattr mlQUZhlcqblpYXqOon[XkeRRf^]V]XgpQjbR]Onp (31) in=40b,setxattr mlhWpPgPW`YhakahTc_fjrboN^dbV\\geTbSTblQl (13680) in=40b,setxattr tnd`PUpci`\\YsrOok\\`[YnRfOdjtnTPaQ\\mmfZad (1754) in=40b,setxattr Ng^``Wsb`a]TmSYVrtjrbplbfeT (1787) in=27b,setxattr dVaqaN^[gldq]kbhcjaZasgTZ[Odhl]aQlfdOYTs (1154) in=40b,setxattr b\\bskO]Slnk\\Z[hXcNpRmeW^Oika`hnp[sjf^gZt (9120) in=40b,setxattr t]^h^V[jTXTXdlVSspZtqdq[YY_sZsYcTSUZ (1787) in=36b,setxattr aahNesb^`gaiRgYcaaOZVfi_pikNPgf\\kp^dVbgb (1997) in=40b,setxattr knq[rqmbh`VtiqaRolRdpfOZYiNUjt_U\\_NNfTiq (31) in=40b,setxattr mbZjYUYt`V_moWdtnSPfObZOZThTNm^fTSbtRmnk (1227) in=40b,setxattr aZashnRSif^YUbVsOs_YajObmqYNc`faT^mn\\ob] (1999) in=40b,setxattr ]VtQgeYWT_YgoePtsWnZO\\`d`b]\\]Oaqnmdlthe\\ (1997) in=40b,setxattr k_teaVkc]RthfgU\\jgWQ`SodpWYqY_nsj\\TqRecZ (66) in=40b,setxattr ropiQcfQSlPeYnYf^msbWfZkoY_m (1787) in=28b,setxattr X]VsnZ]]asTZVtojnVdigiarQe`^U^R^giahQ_la (210) in=40b,setxattr VaaZ\\mfqr^naQSibslaUjPm\\eptUP\\TOeNWaYWS[ (10104) in=40b,setxattr XNmgZfh`iqcZaNiTgXNNZskNflrnOePf__RVdanj (838) in=40b,setxattr nfk[Uja`tYW^hdPlPkRf[n[TeZ\\Oaes`qlh`Vfqp (1505) in=40b,setxattr oQO]kSdpW[PraahiWiph\\lsNqreWq]g]d_rsUVWk (15648) in=40b] snapc 23=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:42:57.375 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.658987+0000", 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.5993205819999998, 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0521995120000001, 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 952 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:42:57.376 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.658987+0000", 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.658987+0000", 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.658990+0000", 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0900000000000001e-06 2024-09-16T04:42:57.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.659096+0000", 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010658900000000001 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.659133+0000", 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6502999999999999e-05 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.378 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.659156+0000", 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3430000000000001e-05 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.402976+0000", 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.743819644 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.403486+0000", 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00050984999999999995 2024-09-16T04:42:57.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.405869+0000", 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0023827919999999999 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.508664+0000", 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10279503700000001 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:42:57.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.710557+0000", 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20189314799999999 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.711186+0000", 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00062942900000000003 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:42:57.381 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:42:57.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:42:57.383 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:42:57.379+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:42:57.484 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:57.479+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:57.504 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:57.505 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:57.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:42:57.585 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:57.578+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:57.611 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:42:57.619 INFO:tasks.rados.rados.0.smithi165.stdout:399: done (2 left) 2024-09-16T04:42:57.619 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 2 2024-09-16T04:42:57.619 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 7 to smithi16530038-37 2024-09-16T04:42:57.619 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 37 updating version 0 to 25 2024-09-16T04:42:57.619 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 37 updating version 25 to 28 2024-09-16T04:42:57.619 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 37 version 28 is already newer than 24 2024-09-16T04:42:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 37 updating version 28 to 29 2024-09-16T04:42:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 37 version 29 is already newer than 26 2024-09-16T04:42:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 37 version 29 is already newer than 27 2024-09-16T04:42:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 29 (ObjNum 143 snap 35 seq_num 143) dirty exists 2024-09-16T04:42:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:396: left oid 37 (ObjNum 143 snap 35 seq_num 143) 2024-09-16T04:42:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:396: done (1 left) 2024-09-16T04:42:57.620 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 1 2024-09-16T04:42:57.644 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:57.645 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:57.645 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:57.645 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:57.645 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:57.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:42:57.686 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:42:57.683+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:122 3.as0 3:5619390f:::smithi16530038-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.105212977999997, 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51059284400000005, 2024-09-16T04:42:57.786 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678088+0000", 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.12e-06 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678098+0000", 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0000000000000001e-05 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678119+0000", 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1676e-05 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678135+0000", 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5526999999999999e-05 2024-09-16T04:42:57.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682759+0000", 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046242649999999998 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682887+0000", 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012783599999999999 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.683968+0000", 2024-09-16T04:42:57.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010808090000000001 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.784795+0000", 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10082761899999999 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188432+0000", 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40363656399999998 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188678+0000", 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024642799999999998 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:123 3.as0 3:5619390f:::smithi16530038-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.105081642000002, 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91283640799999999, 2024-09-16T04:42:57.792 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 123 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678219+0000", 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4030000000000001e-06 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678231+0000", 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678267+0000", 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5719000000000001e-05 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678291+0000", 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4006e-05 2024-09-16T04:42:57.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684378+0000", 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060877320000000002 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684496+0000", 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118076 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188804+0000", 2024-09-16T04:42:57.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50430794400000001 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.189032+0000", 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000227235 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.591053+0000", 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402021666 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:133 3.as0 3:5f4281cc:::smithi16530038-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.899893457000005, 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50755930599999999, 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133 2024-09-16T04:42:57.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883408+0000", 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8669999999999998e-06 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884235+0000", 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082694699999999997 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884270+0000", 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4690000000000002e-05 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884286+0000", 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6082999999999998e-05 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884441+0000", 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155852 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884838+0000", 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000396494 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.886865+0000", 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020266709999999999 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965465+0000", 2024-09-16T04:42:57.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078600544999999994 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390790+0000", 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42532448499999997 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390964+0000", 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017467200000000001 2024-09-16T04:42:57.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:134 3.as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.898981966999997, 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50704782699999995, 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884318+0000", 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884326+0000", 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.32e-06 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884339+0000", 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.274e-05 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884348+0000", 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0229999999999998e-06 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888432+0000", 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040844899999999996 2024-09-16T04:42:57.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888564+0000", 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131419 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889204+0000", 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064011899999999997 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965469+0000", 2024-09-16T04:42:57.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076265854999999994 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391186+0000", 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.425716382 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391364+0000", 2024-09-16T04:42:57.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017849599999999999 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:135 3.as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.898887870999999, 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0075776860000001, 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:42:57.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884412+0000", 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3399999999999997e-07 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884416+0000", 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3059999999999999e-06 2024-09-16T04:42:57.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884428+0000", 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2500000000000001e-05 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884439+0000", 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0662999999999999e-05 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889599+0000", 2024-09-16T04:42:57.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051597179999999998 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889722+0000", 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012330300000000001 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392670+0000", 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50294774399999997 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392905+0000", 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023530799999999999 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.891988+0000", 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49908321 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:176 3.as0 3:5619390f:::smithi16530038-41:head [copy-from ver 10] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.816 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.708601092999999, 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71635610400000005, 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074701+0000", 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9500000000000003e-06 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074716+0000", 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4523e-05 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074742+0000", 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5795999999999998e-05 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074764+0000", 2024-09-16T04:42:57.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2646999999999998e-05 2024-09-16T04:42:57.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074937+0000", 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017243599999999999 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074957+0000", 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9873000000000001e-05 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178498+0000", 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103540929 2024-09-16T04:42:57.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178630+0000", 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132399 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.493151+0000", 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.3145212 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.597069+0000", 2024-09-16T04:42:57.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103917643 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.790921+0000", 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19385244300000001 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791054+0000", 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013226500000000001 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:177 3.as0 3:5619390f:::smithi16530038-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.708481559999996, 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42059460999999998, 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177 2024-09-16T04:42:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074819+0000", 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074828+0000", 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7399999999999993e-06 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074850+0000", 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2802999999999998e-05 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074873+0000", 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2180000000000001e-05 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075093+0000", 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000220645 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075109+0000", 2024-09-16T04:42:57.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5823000000000001e-05 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178898+0000", 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10378908100000001 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178962+0000", 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489000000000001e-05 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495097+0000", 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31613566599999998 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495230+0000", 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133039 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495412+0000", 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018130800000000001 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.829 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:194 3.as0 3:5f4281cc:::smithi16530038-45:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.683322528000005, 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59800688199999996, 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.830 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 194 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099979+0000", 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6400000000000001e-06 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099990+0000", 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.15e-05 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100022+0000", 2024-09-16T04:42:57.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1569999999999998e-05 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100039+0000", 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7486000000000002e-05 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179005+0000", 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078965628999999996 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179189+0000", 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184155 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.486772+0000", 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30758345500000001 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495583+0000", 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088103490000000003 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697796+0000", 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20221325300000001 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697983+0000", 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018684499999999999 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.835 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:276 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 64.684519043999998, 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71405619499999995, 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 276 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098783+0000", 2024-09-16T04:42:57.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098798+0000", 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5213e-05 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098845+0000", 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6853e-05 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098872+0000", 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6622999999999999e-05 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099054+0000", 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018249499999999999 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099075+0000", 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.065e-05 2024-09-16T04:42:57.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207663+0000", 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858786400000001 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207828+0000", 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016534199999999999 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812836+0000", 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60500746100000002 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:277 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 64.684371192, 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81467523799999997, 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:57.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 277 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:42:57.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098929+0000", 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4670000000000001e-06 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098938+0000", 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098964+0000", 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5979999999999999e-05 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098980+0000", 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5917e-05 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099168+0000", 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000188228 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099176+0000", 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.307912+0000", 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20873598700000001 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.308068+0000", 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156448 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913603+0000", 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60553407699999995 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:278 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 64.684295082000006, 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81515018400000006, 2024-09-16T04:42:57.846 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 278 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099005+0000", 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-06 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099012+0000", 2024-09-16T04:42:57.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9e-06 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099031+0000", 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9303000000000001e-05 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099046+0000", 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5003e-05 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099213+0000", 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000166773 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099218+0000", 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408464+0000", 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30924607700000001 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408612+0000", 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014793900000000001 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.914154+0000", 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50554194200000002 2024-09-16T04:42:57.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.as0 43/32 ECSubRead(tid=39, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 64.374378253000003, 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40299000499999998, 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408986+0000", 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409043+0000", 2024-09-16T04:42:57.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6589999999999999e-05 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811767+0000", 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40272446200000001 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811910+0000", 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014330300000000001 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=37, attrs_read=0))", 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 64.373978932, 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.403543547, 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:57.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409389+0000", 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9126000000000002e-05 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409407+0000", 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8097e-05 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812374+0000", 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40296689699999999 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812863+0000", 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048942699999999996 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:42:57.859 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 64.373375697, 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40309429400000002, 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409924+0000", 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409963+0000", 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8753e-05 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409981+0000", 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8737e-05 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409991+0000", 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3889999999999994e-06 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812922+0000", 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40293080100000001 2024-09-16T04:42:57.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.813017+0000", 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5457000000000004e-05 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:704 3.14s0 3:28a14731:::smithi16530038-3:head [rollback d] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.122062504999999, 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40748865899999998, 2024-09-16T04:42:57.863 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 704 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661237+0000", 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2100000000000001e-06 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661243+0000", 2024-09-16T04:42:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0970000000000002e-06 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661259+0000", 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5622999999999999e-05 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661270+0000", 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1182999999999999e-05 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.763754+0000", 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102483703 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.763924+0000", 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000170182 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.765124+0000", 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001200191 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.767658+0000", 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025341579999999999 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068502+0000", 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30084363400000003 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068725+0000", 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000222678 2024-09-16T04:42:57.868 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:705 3.14s0 3:28a14731:::smithi16530038-3:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.121982297999999, 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40781490599999998, 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 705 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661317+0000", 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2729999999999999e-06 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661324+0000", 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3629999999999999e-06 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661335+0000", 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1396999999999999e-05 2024-09-16T04:42:57.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661346+0000", 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0546e-05 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.766069+0000", 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104723092 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.766222+0000", 2024-09-16T04:42:57.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015317900000000001 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.767028+0000", 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080575099999999995 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.868139+0000", 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101110693 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068984+0000", 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20084581600000001 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.069131+0000", 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014679600000000001 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:748 3.14s0 3:2b4d2b94:::smithi16530038-49:head [stat,delete] snapc 19=[19,18,17,16,11,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:42:57.874 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.337936059, 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.406000319, 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 748 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:42:57.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445364+0000", 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445369+0000", 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4140000000000002e-06 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445388+0000", 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8292999999999999e-05 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445402+0000", 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4620000000000001e-05 2024-09-16T04:42:57.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445575+0000", 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017319199999999999 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445594+0000", 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8363e-05 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.651371+0000", 2024-09-16T04:42:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20577725199999999 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.651555+0000", 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018435899999999999 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.653237+0000", 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001681491 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.657570+0000", 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043331439999999997 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.851129+0000", 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19355888900000001 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.851363+0000", 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023383200000000001 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.12s5 97/53 ECSubWrite(tid=35, reqid=osd.9.0:35, at_version=97'36, trim_to=0'0, roll_forward_to=95'34))", 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.224219382999999, 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60339580100000001, 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.58e-07 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559112+0000", 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2073000000000003e-05 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559132+0000", 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0112999999999999e-05 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559148+0000", 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5829e-05 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.060297+0000", 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50114971399999997 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.060328+0000", 2024-09-16T04:42:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0593000000000003e-05 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.162396+0000", 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102068568 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.162475+0000", 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8553000000000005e-05 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.10.22:1 3.14s0 3:2b4d2b94:::smithi16530038-49:head [copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:54.115486+0000", 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.667812273, 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78126567199999997, 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115486+0000", 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115486+0000", 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115489+0000", 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9799999999999998e-06 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115500+0000", 2024-09-16T04:42:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0693000000000001e-05 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115522+0000", 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2419999999999999e-05 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115538+0000", 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5826999999999999e-05 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115722+0000", 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000183542 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115728+0000", 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3260000000000001e-06 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.896273+0000", 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78054450099999995 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.896752+0000", 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000479383 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:980 3.14s0 3:2d731f27:::smithi16530038-30:head [create,setxattr qe_tlRrqOVZN]Sg`OogokNcdi`iTe[kSpgXo[tdi (347) in=40b,setxattr s_\\R^hs]`PhSfjan[pki[NgTNmQrWYik__[]RgPU (594) in=40b,setxattr `RVkQO\\NPp`^id``ktR\\nNYoo[aZVa[a (1401) in=32b,setxattr eUkd\\WaahPacqTYl^iV[kQsis\\aXTQqklkRsYTbs (873) in=40b,setxattr q\\qjarW`i[`bSljr`srl\\Vqni]Z`STgbdTnSmh`o (1963) in=40b,setxattr Sd^`oRTnVWgjdbTpljaQYrhO[``]a^haiQjgUoq` (18032) in=40b,setxattr nUtsh_fZqO^tgbj\\TdjN\\iNYcaa[UnfnknanjssP (1087) in=40b,setxattr _kcTXgOY]Up]^Vac[sbWiZ_aZifkbn`Vphdsslam (1434) in=40b,setxattr ttpkrgNZdj[]rR]iWqO]TaqdilgcW_tR\\^[_Pmgi (1083) in=40b,setxattr gYWdpfi^\\XWiW\\OjsQl_iOetb_TPV\\saa`i]Zkab (593) in=40b,setxattr qijT`_VpNN[TW`kts_Y_ngsTQrrOe_YNONO[^Qla (19224) in=40b,setxattr jXkpPeq`\\r\\hVgrkgqYiY_s[Up`[etXec^VWjblQ (175) in=40b,setxattr Ohgbf\\VQaSpfn^apdUcdOWhk]WUmXU\\`_NcacXQt (139) in=40b,setxattr [egW_laW[O\\dpqaiSbfc]Za^tico``qipqj`Pte` (1050) in=40b,setxattr h\\kdtVlaQPNaZakj[Y`XiYgiflaRcnt\\qnpaNURY (981) in=40b,setxattr as^bS\\_VZOPUeOaf\\]dtfgretlb]]`s``POt_SXc (19508) in=40b,setxattr NQg\\ajVioao__Weae_To_SsOrScUoVTbZRm[oc[O (1297) in=40b,setxattr UZTkha_tgerT`qq\\omSTjZSSXXZQU^f[RWr]`Znn (522) in=40b,setxattr dsdNaS_PtcaUjlnr]mXhVnPO_aZl]RWdh (1401) in=33b,setxattr pNnNhSVlYtVaX`q^i[ZgRflmt\\V_ZpabhitfmqXf (31) in=40b] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:55.704172+0000", 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.0791267320000002, 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.44758898600000002, 2024-09-16T04:42:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 980 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704172+0000", 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704172+0000", 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704175+0000", 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8729999999999999e-06 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704223+0000", 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7966000000000001e-05 2024-09-16T04:42:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704255+0000", 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2870000000000002e-05 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704272+0000", 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6623e-05 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.707855+0000", 2024-09-16T04:42:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035828729999999999 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.708584+0000", 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072875499999999998 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.945732+0000", 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.237148472 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.945947+0000", 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000214489 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.947052+0000", 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011048060000000001 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.948684+0000", 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016326089999999999 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.151033+0000", 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20234907899999999 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.151761+0000", 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072757099999999999 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:57.896 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:57.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:42:57.899 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:57.895+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:57.999 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:57.995+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:58.020 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.10s2 107/102 [PushOp(3:0fc9379d:::smithi16530038-35:1a, version: 97'40, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:0fc9379d:::smithi16530038-35:1a@97'40, size: 2424832, copy_subset: [], clone_subset: {}, snapset: 1a=[]:{1a: [1a, 19]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:57.935602+0000", 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.084693194999999999, 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.084798961000000006, 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "no flag points reached", 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:57.935602+0000", 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:57.935602+0000", 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:57.935603+0000", 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7300000000000004e-07 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:57.935910+0000", 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030692700000000002 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:57.935938+0000", 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T04:42:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-16T04:42:58.024 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:58.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:42:58.068 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:42:58.101 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:42:58.099+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:42:58.203 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:42:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:42:58.203 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:58.203 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:42:58.198+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:42:58.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:42:58.231 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:42:58.232 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:42:58.257 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:42:58.257 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:42:58.278 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:42:58.278 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:42:58.296 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:42:58.296 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:42:58.304 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:58.299+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:58.317 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:42:58.318 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:42:58.340 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:142 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.161735004999997, 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98845964500000005, 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 142 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175559+0000", 2024-09-16T04:42:58.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175562+0000", 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6839999999999998e-06 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.175978+0000", 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041531000000000001 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176014+0000", 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5913e-05 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.176030+0000", 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5872999999999999e-05 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658368+0000", 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.48233846800000002 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.658643+0000", 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000274651 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.659966+0000", 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013230399999999999 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665031+0000", 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050652290000000001 2024-09-16T04:42:58.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.163900+0000", 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.498868961 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164018+0000", 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118516 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:143 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.155256643000001, 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.982808864, 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182037+0000", 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182041+0000", 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5099999999999999e-06 2024-09-16T04:42:58.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182677+0000", 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063672200000000005 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182720+0000", 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2339999999999998e-05 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182742+0000", 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1929999999999998e-05 2024-09-16T04:42:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661153+0000", 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.478411009 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.661298+0000", 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145659 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.662610+0000", 2024-09-16T04:42:58.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001311967 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766105+0000", 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10349428300000001 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164694+0000", 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39858998099999998 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.164846+0000", 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015146300000000001 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:144 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.154516822000005, 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98240055400000004, 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.352 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.353 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.353 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.353 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.353 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 144 2024-09-16T04:42:58.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.353 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182777+0000", 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182779+0000", 2024-09-16T04:42:58.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182787+0000", 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.977e-06 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182810+0000", 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3543000000000001e-05 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182839+0000", 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9e-05 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663652+0000", 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.480813243 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.663795+0000", 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143108 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.664540+0000", 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074409499999999996 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.766192+0000", 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10165231600000001 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165013+0000", 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39882149300000003 2024-09-16T04:42:58.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.165177+0000", 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164146 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:145 3.ds0 3:b6ec1d3e:::smithi16530038-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.154431958999993, 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.385587095, 2024-09-16T04:42:58.358 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182862+0000", 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182863+0000", 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3039999999999999e-06 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182868+0000", 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:42:58.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182891+0000", 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2362999999999999e-05 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.182911+0000", 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9813000000000002e-05 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665100+0000", 2024-09-16T04:42:58.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.482189849 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.665304+0000", 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020365800000000001 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264646+0000", 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59934227699999998 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.264794+0000", 2024-09-16T04:42:58.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147412 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.568449+0000", 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30365514599999999 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.11.21:1 3.ds0 3:bc30766e:::smithi16530038-11:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.124902925000001, 2024-09-16T04:42:58.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0541568889999999, 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212391+0000", 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212394+0000", 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9000000000000002e-06 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212406+0000", 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2289999999999999e-05 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212428+0000", 2024-09-16T04:42:58.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2410000000000001e-05 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.212462+0000", 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3708999999999999e-05 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765550+0000", 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55308802099999999 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.765692+0000", 2024-09-16T04:42:58.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014137199999999999 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.266548+0000", 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50085618700000001 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:172 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.038090988999997, 2024-09-16T04:42:58.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57524237600000006, 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 172 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299203+0000", 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299206+0000", 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6859999999999999e-06 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299276+0000", 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9592999999999996e-05 2024-09-16T04:42:58.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299323+0000", 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6749999999999998e-05 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299347+0000", 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4416000000000001e-05 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469676+0000", 2024-09-16T04:42:58.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.170329287 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.469835+0000", 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159006 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.874445+0000", 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40460963799999999 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:173 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.037906900999999, 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57613007299999996, 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.372 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299387+0000", 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299389+0000", 2024-09-16T04:42:58.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7099999999999999e-06 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299399+0000", 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0866000000000001e-05 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299427+0000", 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7117000000000001e-05 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299488+0000", 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1186000000000007e-05 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470324+0000", 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17083640999999999 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.470484+0000", 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015948199999999999 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.875517+0000", 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40503330199999998 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:174 3.ds0 3:bfd7179d:::smithi16530038-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.037778122000006, 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67607313300000005, 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.376 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299516+0000", 2024-09-16T04:42:58.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299517+0000", 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.393e-06 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299526+0000", 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8230000000000001e-06 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299553+0000", 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6889999999999998e-05 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.299573+0000", 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9944000000000001e-05 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471098+0000", 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.171524865 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.471221+0000", 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012369900000000001 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.975589+0000", 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50436751899999999 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:282 3.ds0 3:b100bb67:::smithi16530038-42:head [copy-from ver 3] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:58.380 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 65.238396346000002, 2024-09-16T04:42:58.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62041838900000001, 2024-09-16T04:42:58.381 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.381 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.381 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.381 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.381 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.381 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 282 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098897+0000", 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098900+0000", 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6630000000000002e-06 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098911+0000", 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0447e-05 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098945+0000", 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4418999999999997e-05 2024-09-16T04:42:58.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098960+0000", 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5119999999999999e-05 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099123+0000", 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016297600000000001 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099133+0000", 2024-09-16T04:42:58.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0125999999999999e-05 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.303007+0000", 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20387381399999999 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.303177+0000", 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016986900000000001 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.615144+0000", 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31196737099999999 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.620953+0000", 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058083869999999999 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.719092+0000", 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098139689000000002 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.719316+0000", 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000223508 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:513 3.ds0 3:b56b9d63:::smithi16530038-2:head [append 0~720896] snapc 10=[f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.108918578999997, 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112358499999997, 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.387 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 513 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228375+0000", 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.228379+0000", 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5269999999999999e-06 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.229180+0000", 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080095700000000004 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.229216+0000", 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6303000000000001e-05 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329324+0000", 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100107881 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329465+0000", 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140796 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.329663+0000", 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019788499999999999 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.332137+0000", 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024749519999999999 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.437411+0000", 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105273296 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839350+0000", 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40193957600000002 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.839499+0000", 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000148412 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:517 3.ds0 3:b56b9d63:::smithi16530038-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:42:58.392 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.006878653000001, 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71224451200000005, 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 517 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330415+0000", 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330416+0000", 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330420+0000", 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.557e-06 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330431+0000", 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1223000000000001e-05 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.330438+0000", 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7170000000000008e-06 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.841180+0000", 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51074163900000003 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.841322+0000", 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141433 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.042660+0000", 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20133805299999999 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:568 3.ds0 3:b100bb67:::smithi16530038-42:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.435780491999999, 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60988380600000003, 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.397 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 568 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901513+0000", 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901535+0000", 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1696999999999999e-05 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901549+0000", 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.364e-05 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901580+0000", 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1335999999999999e-05 2024-09-16T04:42:58.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901639+0000", 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8776000000000001e-05 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.300597+0000", 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.398958549 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.300848+0000", 2024-09-16T04:42:58.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025097800000000001 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.302460+0000", 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016119750000000001 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.306368+0000", 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039082070000000003 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.511231+0000", 2024-09-16T04:42:58.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.204862715 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.511397+0000", 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000165933 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:569 3.ds0 3:b100bb67:::smithi16530038-42:head [rollback 11] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.435544180999997, 2024-09-16T04:42:58.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80813885399999996, 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 569 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901750+0000", 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901751+0000", 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6670000000000001e-06 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901804+0000", 2024-09-16T04:42:58.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2306999999999999e-05 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901826+0000", 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2186000000000001e-05 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901845+0000", 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8802999999999999e-05 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.409423+0000", 2024-09-16T04:42:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.507578164 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.409660+0000", 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023717500000000001 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.410803+0000", 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011429249999999999 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.611780+0000", 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20097747999999999 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.709760+0000", 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097980190999999994 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.709888+0000", 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012795600000000001 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:570 3.ds0 3:b100bb67:::smithi16530038-42:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.435419844999998, 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81140166199999997, 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 570 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.408 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901874+0000", 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901875+0000", 2024-09-16T04:42:58.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.483e-06 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901883+0000", 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.357e-06 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901901+0000", 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8593e-05 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:18.901917+0000", 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.556e-05 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.509597+0000", 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60768048500000005 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.509774+0000", 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000176909 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.510566+0000", 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079207799999999999 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.611782+0000", 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101216059 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.713154+0000", 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10137199199999999 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:19.713276+0000", 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121146 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:754 3.ds0 3:b743ddf6:::smithi16530038-25:head [setxattr _header (57) in=7b] snapc 19=[19,18,17,16,11,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:36.450830+0000", 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.886463631000002, 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70760600399999996, 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.413 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 754 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450830+0000", 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450830+0000", 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450831+0000", 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2240000000000001e-06 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450837+0000", 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7100000000000004e-06 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450851+0000", 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4156e-05 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450861+0000", 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9769999999999995e-06 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.658220+0000", 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20735928200000001 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.658396+0000", 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017554499999999999 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.659362+0000", 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00096583300000000003 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.662442+0000", 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030804130000000002 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158286+0000", 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49584404500000001 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158436+0000", 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014981900000000001 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.418 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:755 3.ds0 3:b743ddf6:::smithi16530038-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:36.450893+0000", 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.886400541, 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71187130499999995, 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 755 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450893+0000", 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450893+0000", 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450894+0000", 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9300000000000006e-07 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450898+0000", 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1069999999999998e-06 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450911+0000", 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2799999999999999e-05 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450920+0000", 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.659849+0000", 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20892882600000001 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.659990+0000", 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141056 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158659+0000", 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49866967000000001 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.158953+0000", 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029351399999999998 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.162765+0000", 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003811629 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.423 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:756 3.ds0 3:b100bb67:::smithi16530038-42:head [stat,delete] snapc 19=[19,18,17,16,11,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:36.450932+0000", 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.886361421, 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70945773899999998, 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 756 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450932+0000", 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450932+0000", 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450933+0000", 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2699999999999998e-07 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450937+0000", 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6430000000000001e-06 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450950+0000", 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2989999999999999e-05 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.450957+0000", 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4460000000000004e-06 2024-09-16T04:42:58.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.660125+0000", 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.209168248 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.660286+0000", 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016025300000000001 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.661434+0000", 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011481040000000001 2024-09-16T04:42:58.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:36.663261+0000", 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018268659999999999 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.160204+0000", 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49694368999999999 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:37.160390+0000", 2024-09-16T04:42:58.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018567200000000001 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(3.1ds0 101/98 [PushReplyOp(3:bc0e3a40:::smithi16530038-18:head)])", 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:52.362059+0000", 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9752351140000002, 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66418297900000001, 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.362059+0000", 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.362059+0000", 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.362060+0000", 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.866e-06 2024-09-16T04:42:58.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.362071+0000", 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0679999999999999e-05 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.362116+0000", 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5136999999999998e-05 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.362133+0000", 2024-09-16T04:42:58.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6433e-05 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.025997+0000", 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66386467800000004 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.026242+0000", 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024418500000000001 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(3.1ds0 101/98 [PushReplyOp(3:bc0e3a40:::smithi16530038-18:head)])", 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:52.446859+0000", 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.8904343739999998, 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.582936287, 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.446859+0000", 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.446859+0000", 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.446861+0000", 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.677e-06 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.446871+0000", 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5629999999999998e-06 2024-09-16T04:42:58.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.446900+0000", 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9853e-05 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.446918+0000", 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7076e-05 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.029699+0000", 2024-09-16T04:42:58.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.58278127300000004 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.029796+0000", 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6844999999999999e-05 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(3.1ds0 101/98 [PushReplyOp(3:bc0e3a40:::smithi16530038-18:head)])", 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:52.461649+0000", 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.8756446750000002, 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.56831112699999997, 2024-09-16T04:42:58.436 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.461649+0000", 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.461649+0000", 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.461651+0000", 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9829999999999999e-06 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.461662+0000", 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0774000000000001e-05 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.461692+0000", 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0589000000000001e-05 2024-09-16T04:42:58.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.461707+0000", 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5063e-05 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.029870+0000", 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.56816248800000002 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.029960+0000", 2024-09-16T04:42:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0229999999999995e-05 2024-09-16T04:42:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:42:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:42:58.440 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:58.441 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:42:58.442 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:42:58.442 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:42:58.442 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:42:58.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:42:58.443 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:58.434+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:58.470 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:42:58.470 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:42:58.542 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:58.538+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:58.569 INFO:teuthology.orchestra.run.smithi165.stdout: marking collection for removal 2024-09-16T04:42:58.570 INFO:teuthology.orchestra.run.smithi165.stdout:setting '_remove' omap key 2024-09-16T04:42:58.570 INFO:teuthology.orchestra.run.smithi165.stdout:finish_remove_pgs 3.2s4_head removing 3.2s4 2024-09-16T04:42:58.570 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:42:58.570 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:42:58.571 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:42:58.571 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:42:58.575 INFO:teuthology.orchestra.run.smithi165.stdout:Remove successful 2024-09-16T04:42:58.580 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:42:58.581 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:42:58.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:42:58.619 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:42:58.619 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:42:58.643 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:58.638+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:58.658 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:42:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:42:58.692 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:42:58.692 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:42:58.703 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:58.703 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:58.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:42:58.721 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:42:58.721 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:42:58.744 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:42:58.738+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:42:58.756 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:42:58.756 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:42:58.784 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:42:58.785 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:42:58.785 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.15 2024-09-16T04:42:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:42:58.790 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:42:58.842 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:58.843 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:58.845 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:42:58.843+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:42:58.858 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:42:58.946 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:58.942+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:58.997 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:58.997 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:59.012 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:42:59.047 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:59.043+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:59.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-15 --journal-path /var/lib/ceph/osd/ceph-15/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.2s4.15 2024-09-16T04:42:59.148 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:42:59.143+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:42:59.180 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:59.180 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:59.195 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:42:59.249 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:59.243+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:59.323 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:59.323 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:59.337 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:42:59.350 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:42:59.342+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:42:59.451 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:42:59.446+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:42:59.457 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:59.458 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:59.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:42:59.551 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:42:59.547+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:42:59.643 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:42:59.647 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:59.647 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:59.665 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:42:59.665 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:42:59.658+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:42:59.753 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:42:59.750+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:42:59.847 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:59.847 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:59.853 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:42:59.846+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:42:59.862 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:42:59.955 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:42:59.951+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:42:59.984 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:42:59.984 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:42:59.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:43:00.056 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:00.050+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:00.116 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:00.116 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:00.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:43:00.143 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:43:00.157 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:00.155+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:00.251 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:00.251 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:00.258 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:00.255+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:00.268 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:43:00.359 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:43:00.355+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:43:00.386 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:00.386 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:00.401 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:43:00.460 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:00.455+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:00.523 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:00.523 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:00.538 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:43:00.582 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:43:00.578+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:43:00.653 INFO:teuthology.orchestra.run.smithi165.stdout:get_pg_num_history pg_num_history pg_num_history(e99 pg_nums {1={22=1},2={25=8},3={32=16,37=26,47=25,51=24,55=26,67=36}} deleted_pools ) 2024-09-16T04:43:00.653 INFO:teuthology.orchestra.run.smithi165.stdout:Importing pgid 3.2s4 2024-09-16T04:43:00.654 INFO:teuthology.orchestra.run.smithi165.stdout:write_pg epoch 97 info 3.2s4( v 59'15 (0'0,59'15] local-lis/les=67/68 n=1 ec=32/32 lis/c=67/67 les/c/f=68/68/0 sis=67) 2024-09-16T04:43:00.656 INFO:teuthology.orchestra.run.smithi165.stdout:Import successful 2024-09-16T04:43:00.662 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:43:00.659+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:43:00.666 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:00.666 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:00.681 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:43:00.763 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:00.759+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:00.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:43:00.813 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:00.813 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:00.828 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:43:00.863 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:00.859+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:00.947 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:43:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:43:00.950 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:00.950 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:00.964 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:43:00.965 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:00.958+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:00.968 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:43:00.968 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:43:00.992 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:43:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:43:01.015 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:43:01.015 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:43:01.038 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:43:01.038 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:43:01.060 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:43:01.060 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:43:01.064 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:43:01.059+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:43:01.083 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:43:01.084 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:43:01.086 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:43:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:43:01.086 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 36/32 ECSubWrite(tid=10, reqid=client.4728.0:122, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 75.398898853000006, 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201524707, 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:43:01.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684050+0000", 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684051+0000", 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.277e-06 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684081+0000", 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0414999999999999e-05 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684109+0000", 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7455000000000001e-05 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.684125+0000", 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5707000000000001e-05 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884504+0000", 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200379487 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.884519+0000", 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4953000000000001e-05 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885504+0000", 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098537600000000005 2024-09-16T04:43:01.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.885574+0000", 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0036999999999994e-05 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=36'2))", 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 75.108577253999997, 2024-09-16T04:43:01.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20260964000000001, 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974371+0000", 2024-09-16T04:43:01.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974373+0000", 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.683e-06 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974841+0000", 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468255 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:45.974891+0000", 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0198000000000003e-05 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.074965+0000", 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100073226 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175170+0000", 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100205119 2024-09-16T04:43:01.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.175184+0000", 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4626e-05 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176898+0000", 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017136390000000001 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.176981+0000", 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2893999999999996e-05 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/32 ECSubWrite(tid=15, reqid=client.4728.0:160, at_version=36'10, trim_to=0'0, roll_forward_to=36'7))", 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 74.629285644000007, 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20191688999999999, 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.096 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453663+0000", 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453664+0000", 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453695+0000", 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0998999999999999e-05 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.453714+0000", 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9745000000000002e-05 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.553888+0000", 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10017351100000001 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654137+0000", 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100248949 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.654151+0000", 2024-09-16T04:43:01.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4279e-05 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655526+0000", 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013750520000000001 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.655580+0000", 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3634999999999997e-05 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 48/32 ECSubWrite(tid=44, reqid=client.4728.0:362, at_version=48'17, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 62.726262343999998, 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205456467, 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.101 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356686+0000", 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356687+0000", 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.356985+0000", 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029765100000000001 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357014+0000", 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9853e-05 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.357034+0000", 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9592000000000001e-05 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557308+0000", 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20027399500000001 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.557322+0000", 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4346e-05 2024-09-16T04:43:01.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562072+0000", 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047490960000000004 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:58.562143+0000", 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1191000000000003e-05 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.105 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s6 57/53 ECSubWrite(tid=9, reqid=osd.10.0:9, at_version=57'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 52.853831436999997, 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209615832, 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229117+0000", 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229119+0000", 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229208+0000", 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9190999999999994e-05 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.229249+0000", 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0842e-05 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.329348+0000", 2024-09-16T04:43:01.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10009905600000001 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426747+0000", 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.09739892 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.426767+0000", 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0173000000000002e-05 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438659+0000", 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011892171 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.438733+0000", 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3515000000000003e-05 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s3 63/55 ECSubWrite(tid=22, reqid=osd.5.0:22, at_version=63'14, trim_to=0'0, roll_forward_to=62'12))", 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:43:01.110 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 47.504552296, 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20251555199999999, 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578396+0000", 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9999999999999999e-07 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578420+0000", 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3271000000000001e-05 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.578449+0000", 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9488999999999998e-05 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.678595+0000", 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100145814 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778824+0000", 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100228703 2024-09-16T04:43:01.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.778840+0000", 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6123000000000001e-05 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780838+0000", 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019980129999999999 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:13.780912+0000", 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3938999999999998e-05 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es2 69/67 ECSubWrite(tid=80, reqid=client.4728.0:581, at_version=0'0, trim_to=0'0, roll_forward_to=69'54))", 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 41.356377582999997, 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20154364599999999, 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.115 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726571+0000", 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726573+0000", 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1399999999999998e-06 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726628+0000", 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4376000000000003e-05 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.726668+0000", 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0466000000000002e-05 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.826832+0000", 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100163697 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827016+0000", 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018461700000000001 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.827048+0000", 2024-09-16T04:43:01.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2186e-05 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928006+0000", 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10095765900000001 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:19.928115+0000", 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010850500000000001 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1es3 74/69 ECSubWrite(tid=62, reqid=client.4728.0:620, at_version=74'53, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 36.099098421000001, 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20195012000000001, 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:43:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983850+0000", 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983851+0000", 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.173e-06 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983905+0000", 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4027000000000003e-05 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.983927+0000", 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2005999999999999e-05 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084064+0000", 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10013664899999999 2024-09-16T04:43:01.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084267+0000", 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020252399999999999 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.084288+0000", 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1943999999999999e-05 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.185701+0000", 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101412144 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:25.185800+0000", 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9653000000000003e-05 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:697 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~704512] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:01.124 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.592042902999999, 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.26883937299999999, 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 697 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.125 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490906+0000", 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.490907+0000", 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8050000000000001e-06 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491858+0000", 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095084100000000001 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491869+0000", 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1184e-05 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491878+0000", 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3480000000000004e-06 2024-09-16T04:43:01.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491963+0000", 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5368e-05 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491974+0000", 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1161e-05 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492735+0000", 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00076025099999999998 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492939+0000", 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020470800000000001 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.494899+0000", 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019597260000000002 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.496840+0000", 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0019404660000000001 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759668+0000", 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.262828544 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759745+0000", 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6971000000000006e-05 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.130 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:698 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~655360] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.591047943, 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.36985094099999999, 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.131 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 698 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491901+0000", 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.491902+0000", 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9100000000000002e-07 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492632+0000", 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00073060200000000001 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.492643+0000", 2024-09-16T04:43:01.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0910999999999999e-05 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592713+0000", 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100069806 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593257+0000", 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00054396700000000002 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593384+0000", 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012739399999999999 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.594523+0000", 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011388450000000001 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759798+0000", 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.16527468300000001 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.861671+0000", 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101873753 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.861752+0000", 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0088999999999998e-05 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:699 3.1fs0 3:fb308b36:::smithi16530038-23:head [append 0~507904] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.490205286999998, 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.37090961900000002, 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 699 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.137 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592743+0000", 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.592745+0000", 2024-09-16T04:43:01.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7e-06 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593160+0000", 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041519899999999999 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593172+0000", 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2248e-05 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.139 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593182+0000", 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4210000000000001e-06 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.595327+0000", 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0021456460000000002 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.595409+0000", 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1316000000000005e-05 2024-09-16T04:43:01.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.596547+0000", 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011381710000000001 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759799+0000", 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.163252014 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:01.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963555+0000", 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203755827 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963653+0000", 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8077000000000003e-05 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:700 3.1fs0 3:fb308b36:::smithi16530038-23:head [setxattr _header (57) in=7b] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:43:01.142 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.489746926999999, 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.37068284299999998, 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 700 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.143 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593202+0000", 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6700000000000003e-07 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593207+0000", 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7740000000000004e-06 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593217+0000", 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9329999999999996e-06 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593225+0000", 2024-09-16T04:43:01.145 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4749999999999996e-06 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.597569+0000", 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0043444690000000001 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.597710+0000", 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141125 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.146 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.598396+0000", 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000685767 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.759801+0000", 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.16140457599999999 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963785+0000", 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20398479 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.147 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.963884+0000", 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9166999999999994e-05 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:701 3.1fs0 3:fb308b36:::smithi16530038-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.489702878999999, 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.67642940299999998, 2024-09-16T04:43:01.148 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 701 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593246+0000", 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.51e-07 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593250+0000", 2024-09-16T04:43:01.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0099999999999997e-06 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593260+0000", 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6360000000000006e-06 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.593267+0000", 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2869999999999999e-06 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.600494+0000", 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072266040000000002 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.600669+0000", 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000174799 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.964260+0000", 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.36359097400000001 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.964360+0000", 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000100375 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.269675+0000", 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.305315167 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s3 79/53 ECSubWrite(tid=126, reqid=client.4728.0:704, at_version=79'66, trim_to=0'0, roll_forward_to=72'63))", 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.318169280999999, 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20206831, 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764779+0000", 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764781+0000", 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.796e-06 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764820+0000", 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8662999999999999e-05 2024-09-16T04:43:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.764850+0000", 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0349999999999999e-05 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.864962+0000", 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100111618 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965141+0000", 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100179383 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965156+0000", 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4997000000000001e-05 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.966786+0000", 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016294300000000001 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.966848+0000", 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2073e-05 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1fs0 79/69 ECSubWriteReply(tid=4, last_complete=79'56, committed=1, applied=1))", 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.117312513000002, 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202173098, 2024-09-16T04:43:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965636+0000", 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965637+0000", 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9400000000000005e-07 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965640+0000", 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4130000000000002e-06 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965650+0000", 2024-09-16T04:43:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0207000000000001e-05 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965659+0000", 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6759999999999996e-06 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167760+0000", 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202100745 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167809+0000", 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9462999999999997e-05 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1fs0 79/69 ECSubWriteReply(tid=4, last_complete=79'56, committed=1, applied=1))", 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.117173734000001, 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202105168, 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965775+0000", 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965777+0000", 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6700000000000001e-06 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965789+0000", 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2173e-05 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965816+0000", 2024-09-16T04:43:01.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7270000000000001e-05 2024-09-16T04:43:01.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.965837+0000", 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.107e-05 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167854+0000", 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20201680599999999 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.167880+0000", 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6179000000000001e-05 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.169 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1fs0 79/69 ECSubRead(tid=5, to_read={3:fb308b36:::smithi16530038-23:head=0,4096,0}, subchunks={3:fb308b36:::smithi16530038-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.115748633999999, 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20164594899999999, 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967200+0000", 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:29.967275+0000", 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4894999999999994e-05 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.067391+0000", 2024-09-16T04:43:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100115909 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.168332+0000", 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100941066 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.168846+0000", 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051407899999999997 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as2 80/67 ECSubWrite(tid=131, reqid=osd.3.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=80'51))", 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 30.385785010999999, 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30121982800000002, 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:43:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697164+0000", 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697165+0000", 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.623e-06 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697185+0000", 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0103e-05 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697215+0000", 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9733000000000001e-05 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.697231+0000", 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5699999999999999e-05 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.897603+0000", 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20037271700000001 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.897620+0000", 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6983999999999999e-05 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.998295+0000", 2024-09-16T04:43:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100674176 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.998383+0000", 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8792e-05 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPush(3.19s6 95/58 [PushOp(3:9de467d5:::smithi16530038-27:7, version: 50'27, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:9de467d5:::smithi16530038-27:7@50'27, size: 1294336, copy_subset: [], clone_subset: {}, snapset: 11=[]:{4: [3], 7: [7], 11: [11, d, b]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:43:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:46.633350+0000", 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 14.449598271999999, 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205371676, 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633350+0000", 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633350+0000", 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633351+0000", 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3399999999999999e-07 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633649+0000", 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029821699999999998 2024-09-16T04:43:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.633711+0000", 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2002999999999996e-05 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.733901+0000", 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100189615 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.837764+0000", 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10386297899999999 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.838722+0000", 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095832800000000004 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1es2 100/97 ECSubWrite(tid=188, reqid=client.4728.0:926, at_version=0'0, trim_to=0'0, roll_forward_to=99'92))", 2024-09-16T04:43:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:51.288513+0000", 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 9.7944356159999995, 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20181585499999999, 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288513+0000", 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288513+0000", 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288515+0000", 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7e-06 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288545+0000", 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9865999999999998e-05 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.288575+0000", 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0329999999999999e-05 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.388711+0000", 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10013585799999999 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.489095+0000", 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10038437 2024-09-16T04:43:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.489112+0000", 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7022999999999999e-05 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.490243+0000", 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001130806 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:51.490329+0000", 2024-09-16T04:43:01.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5902000000000005e-05 2024-09-16T04:43:01.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.187 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.187 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.187 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:01.188 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:43:01.188 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:43:01.189 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:43:01.189 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:43:01.190 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:43:01.190 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:43:01.190 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:43:01.191 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:43:01.191 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:43:01.192 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:43:01.183+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:43:01.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /var/log/ceph/exp.3.2s4.15 2024-09-16T04:43:01.207 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:43:01.207 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:43:01.228 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:43:01.228 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:43:01.251 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:43:01.251 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:43:01.258 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.15 2024-09-16T04:43:01.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-15 show-label 2024-09-16T04:43:01.276 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:43:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:43:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.15 2024-09-16T04:43:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:43:01.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:43:01.322 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:01.322 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:43:01.322 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:43:01.322 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:01.326 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:01.322+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:01.336 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:43:01.337 INFO:teuthology.orchestra.run.smithi165.stdout:inferring bluefs devices from bluestore path 2024-09-16T04:43:01.391 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:43:01.387+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:43:01.462 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:01.462 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:43:01.462 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:43:01.462 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:43:01.462 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:01.479 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:43:01.492 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:01.487+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:01.593 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:01.587+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:01.601 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:01.601 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:43:01.601 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:43:01.601 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4356.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 94.200143948000004, 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31207237399999999, 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4356", 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.59:0/2607617674", 2024-09-16T04:43:01.602 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398124+0000", 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.603 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398126+0000", 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0999999999999998e-06 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398137+0000", 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0253e-05 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398158+0000", 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1401999999999999e-05 2024-09-16T04:43:01.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.398176+0000", 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8552000000000001e-05 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597014+0000", 2024-09-16T04:43:01.605 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19883803 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.597043+0000", 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8683999999999998e-05 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.604764+0000", 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0077206829999999999 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605002+0000", 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023771400000000001 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 15", 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.605234+0000", 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023226099999999999 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.606579+0000", 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013450650000000001 2024-09-16T04:43:01.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710048+0000", 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.103468869 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710127+0000", 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9491000000000002e-05 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.608 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:27.710197+0000", 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9270000000000006e-05 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:190 3.8s0 3:16f2ba44:::smithi16530038-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.498159771000005, 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70466111799999998, 2024-09-16T04:43:01.609 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 190 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.610 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100108+0000", 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100109+0000", 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7300000000000004e-07 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.611 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100116+0000", 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3099999999999997e-06 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100128+0000", 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2262000000000001e-05 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.100137+0000", 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8699999999999998e-06 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.612 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480895+0000", 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.38075793800000002 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.480902+0000", 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0700000000000001e-06 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582260+0000", 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101357841 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.613 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.582314+0000", 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4175000000000002e-05 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804615+0000", 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.22230154099999999 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804681+0000", 2024-09-16T04:43:01.614 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5356000000000002e-05 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.804769+0000", 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8782000000000005e-05 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.615 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 69.188894985000005, 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.311529638, 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.616 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409373+0000", 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409375+0000", 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8899999999999999e-06 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409439+0000", 2024-09-16T04:43:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4428e-05 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409480+0000", 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0454999999999998e-05 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.409505+0000", 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4620000000000001e-05 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619957+0000", 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21045218900000001 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.619979+0000", 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1866e-05 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720830+0000", 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100851396 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.619 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720903+0000", 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2793999999999995e-05 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.10s4 42/41 ECSubWrite(tid=31, reqid=client.4728.0:186, at_version=42'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 69.183610110000004, 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.306308988, 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.620 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414658+0000", 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414660+0000", 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414693+0000", 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3071999999999998e-05 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414716+0000", 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3356000000000001e-05 2024-09-16T04:43:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.414734+0000", 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7787000000000001e-05 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621156+0000", 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.206422364 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.621180+0000", 2024-09-16T04:43:01.623 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3589999999999999e-05 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720911+0000", 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099731400999999997 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:52.720967+0000", 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5841000000000003e-05 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.624 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.8s0 47/37 ECSubReadReply(tid=15, attrs_read=0))", 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 64.233889266000006, 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30496093299999999, 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:43:01.625 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364379+0000", 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364381+0000", 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.663e-06 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.626 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364619+0000", 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023814999999999999 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364642+0000", 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2875999999999999e-05 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.364655+0000", 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3849000000000001e-05 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.627 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.667942+0000", 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30328671699999998 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:57.669340+0000", 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013976780000000001 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.628 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:523 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.365146856999999, 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41544228799999999, 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 523 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.629 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233121+0000", 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233124+0000", 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9069999999999999e-06 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.630 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233135+0000", 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1303e-05 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233167+0000", 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1415999999999999e-05 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233183+0000", 2024-09-16T04:43:01.631 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6495999999999999e-05 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.239234+0000", 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006050153 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.239492+0000", 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000258872 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.632 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.648564+0000", 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40907114100000003 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:524 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.365058015999999, 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41687002699999998, 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.633 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 524 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.634 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233210+0000", 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233211+0000", 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0100000000000001e-06 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233233+0000", 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1647000000000001e-05 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.635 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233246+0000", 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.332e-05 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233256+0000", 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7319999999999993e-06 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.240876+0000", 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076197690000000002 2024-09-16T04:43:01.636 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241108+0000", 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000232103 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.650080+0000", 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40897244599999999 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.637 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:525 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.364995112999999, 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41781336400000002, 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 525 2024-09-16T04:43:01.638 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233273+0000", 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.639 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233274+0000", 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1999999999999998e-07 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233279+0000", 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6799999999999998e-06 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233293+0000", 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3169e-05 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.640 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.233303+0000", 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0540000000000001e-05 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.343879+0000", 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11057620899999999 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344065+0000", 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000186083 2024-09-16T04:43:01.641 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.651086+0000", 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30702086299999998 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.17s0 65/53 ECSubRead(tid=25, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.356631872999998, 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30542288899999998, 2024-09-16T04:43:01.642 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.643 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241636+0000", 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241846+0000", 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021020300000000001 2024-09-16T04:43:01.644 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.241873+0000", 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6367e-05 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.546861+0000", 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30498862300000001 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.547059+0000", 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197696 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 64/53 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.349333682000001, 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.39834076099999999, 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.646 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248934+0000", 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.647 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.248935+0000", 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249102+0000", 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166933 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249127+0000", 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5055999999999999e-05 2024-09-16T04:43:01.648 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.249156+0000", 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8572999999999999e-05 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647206+0000", 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.39805037799999998 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.649 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647275+0000", 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8837999999999994e-05 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.17s0 65/53 ECSubReadReply(tid=25, attrs_read=0))", 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.256426167999997, 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30554653199999998, 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.650 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341842+0000", 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.341843+0000", 2024-09-16T04:43:01.651 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.533e-06 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342145+0000", 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00030105099999999999 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342175+0000", 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0159e-05 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.652 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.342192+0000", 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7003e-05 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647333+0000", 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30514099700000002 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647388+0000", 2024-09-16T04:43:01.653 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5788999999999997e-05 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.17s0 65/53 ECSubRead(tid=26, to_read={3:ed82f61a:::smithi16530038-1:head=0,151552,0}, subchunks={3:ed82f61a:::smithi16530038-1:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.253793510000001, 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30318800499999998, 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.654 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:43:01.655 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344475+0000", 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344537+0000", 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2811999999999994e-05 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.656 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.344572+0000", 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4798999999999997e-05 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647490+0000", 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30291773100000002 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:15.647663+0000", 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017266300000000001 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.657 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPushReply(3.1ds0 74/70 [PushReplyOp(3:bfd7179d:::smithi16530038-33:c)])", 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 37.09903035, 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30809602800000002, 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.658 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499238+0000", 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1399999999999997e-07 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.659 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499244+0000", 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8300000000000001e-06 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499262+0000", 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7346e-05 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499275+0000", 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3376e-05 2024-09-16T04:43:01.660 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807219+0000", 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30794440000000001 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807334+0000", 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000114462 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.661 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPushReply(3.1ds0 74/70 [PushReplyOp(3:bfd7179d:::smithi16530038-33:c)])", 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 37.098840471000003, 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30806645799999999, 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:43:01.662 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499428+0000", 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.46e-07 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.663 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499433+0000", 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5140000000000003e-06 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499446+0000", 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2918999999999999e-05 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.499456+0000", 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0093e-05 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.664 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807424+0000", 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30796800600000002 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807494+0000", 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0480000000000003e-05 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.665 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPushReply(3.1ds0 74/70 [PushReplyOp(3:bfd7179d:::smithi16530038-33:c)])", 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:24.500211+0000", 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 37.098057038999997, 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30692697899999999, 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500211+0000", 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.666 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500211+0000", 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500212+0000", 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.13e-07 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500217+0000", 2024-09-16T04:43:01.667 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5799999999999999e-06 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500230+0000", 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3200000000000001e-05 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.500248+0000", 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7402999999999999e-05 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.668 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807054+0000", 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30680581000000001 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:24.807138+0000", 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4373e-05 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:721 3.1ds0 3:bc0e3a40:::smithi16530038-18:head [append 0~737280] snapc 16=[16,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T04:43:01.669 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:31.461810+0000", 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 30.136458153, 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61923460500000005, 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 721 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.670 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.461810+0000", 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.461810+0000", 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.461813+0000", 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.123e-06 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.671 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.463204+0000", 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013910960000000001 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.463242+0000", 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8083000000000001e-05 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.563407+0000", 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100164278 2024-09-16T04:43:01.672 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.563612+0000", 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020560500000000001 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.563630+0000", 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8343e-05 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.865484+0000", 2024-09-16T04:43:01.673 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30185366200000002 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.865675+0000", 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019052500000000001 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.868329+0000", 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026546040000000001 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.674 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.874291+0000", 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0059619089999999996 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:32.080892+0000", 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.206600961 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:32.081045+0000", 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000152416 2024-09-16T04:43:01.675 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:742 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:35.444909+0000", 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.153359314999999, 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40172697000000002, 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.676 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 742 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444909+0000", 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444909+0000", 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.677 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444912+0000", 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7659999999999999e-06 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444925+0000", 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3196999999999999e-05 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.678 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444958+0000", 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3160000000000001e-05 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.444974+0000", 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6447e-05 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445141+0000", 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016653100000000001 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445164+0000", 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3499999999999999e-05 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.543056+0000", 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097891299000000001 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.543207+0000", 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015134299999999999 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.680 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.846636+0000", 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30342872700000001 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:743 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:35.445011+0000", 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.153257365999998, 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.40293953599999999, 2024-09-16T04:43:01.681 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 743 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445011+0000", 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.682 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445011+0000", 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445012+0000", 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.283e-06 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445020+0000", 2024-09-16T04:43:01.683 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2029999999999992e-06 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445044+0000", 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3354e-05 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445060+0000", 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5973000000000001e-05 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.684 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445286+0000", 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022647100000000001 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445297+0000", 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.097e-05 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.543835+0000", 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.098537793999999998 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.685 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.544002+0000", 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000167248 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.847950+0000", 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394823999999998 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:744 3.17s0 3:ed82f61a:::smithi16530038-1:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:35.445135+0000", 2024-09-16T04:43:01.686 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.153133394000001, 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.403995469, 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 744 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445135+0000", 2024-09-16T04:43:01.687 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445135+0000", 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445137+0000", 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9800000000000001e-06 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445147+0000", 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.058e-05 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.445166+0000", 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9110000000000002e-05 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.545261+0000", 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100094422 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.689 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.545385+0000", 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012448200000000001 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.545629+0000", 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024355700000000001 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.690 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:35.849130+0000", 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30350133800000001 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:966 3.8s0 3:16f2ba44:::smithi16530038-21:head [stat,delete] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:54.595772+0000", 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.002495959, 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.418492266, 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:01.691 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 966 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.595772+0000", 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.692 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.595772+0000", 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.595775+0000", 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.39e-06 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.595784+0000", 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1330000000000005e-06 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.693 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.595814+0000", 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0151999999999999e-05 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.595830+0000", 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6198999999999999e-05 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.696035+0000", 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10020475399999999 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.694 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.696050+0000", 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5310000000000001e-05 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.901595+0000", 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.205544529 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.901600+0000", 2024-09-16T04:43:01.695 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3600000000000004e-06 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.909408+0000", 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0078082990000000003 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.909620+0000", 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021208300000000001 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.696 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.910461+0000", 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00084073399999999999 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.011992+0000", 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101530726 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.014077+0000", 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020853820000000002 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.697 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.014264+0000", 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018721499999999999 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.698 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:01.700 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:43:01.700 INFO:teuthology.orchestra.run.smithi165.stdout: "/var/lib/ceph/osd/ceph-15/block": { 2024-09-16T04:43:01.700 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_uuid": "8335941d-9563-4527-90f6-3b7f9129f530", 2024-09-16T04:43:01.700 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 96636764160, 2024-09-16T04:43:01.700 INFO:teuthology.orchestra.run.smithi165.stdout: "btime": "2024-09-16T04:40:58.861852+0000", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "main", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "bfm_blocks": "23592960", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "bfm_blocks_per_key": "128", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "bfm_bytes_per_block": "4096", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "bfm_size": "96636764160", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs": "1", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_fsid": "bdbec34e-c8aa-4776-9d32-ea336a99655f", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version_when_created": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "created_at": "2024-09-16T04:41:01.450471Z", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "elastic_shared_blobs": "1", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": "16", 2024-09-16T04:43:01.701 INFO:teuthology.orchestra.run.smithi165.stdout: "kv_backend": "rocksdb", 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout: "magic": "ceph osd volume v026", 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout: "multi": "yes", 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout: "ready": "ready", 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout: "require_osd_release": "19", 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "bluestore", 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout: "whoami": "15" 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:01.702 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:43:01.702 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:43:01.703 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.15 2024-09-16T04:43:01.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-15 show-sharding 2024-09-16T04:43:01.704 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:43:01.694+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:43:01.800 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:01.795+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:01.834 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.3s3 111/110 ECSubRead(tid=268, to_read={3:c31e3ec6:::smithi16530038-50:head=0,962560,0}, subchunks={3:c31e3ec6:::smithi16530038-50:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:43:01.771066+0000", 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.063282694, 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.063331849999999995, 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "no flag points reached", 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:43:01.771066+0000", 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:43:01.771066+0000", 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:43:01.771066+0000", 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2e-07 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:43:01.771073+0000", 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1720000000000001e-06 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:43:01.771085+0000", 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1559999999999999e-05 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2024-09-16T04:43:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 1 2024-09-16T04:43:01.838 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:01.849 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:43:01.901 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:43:01.894+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:43:01.977 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:01.977 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:43:01.977 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:43:01.977 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:43:01.977 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:01.992 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:43:02.002 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:01.999+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:02.098 INFO:tasks.rados.rados.0.smithi165.stdout:398: expect (ObjNum 120 snap 24 seq_num 120) 2024-09-16T04:43:02.103 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:43:02.099+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:43:02.168 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 36/32 ECSubWrite(tid=11, reqid=client.4728.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 76.077891023000006, 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20239900399999999, 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087077+0000", 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087079+0000", 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5730000000000002e-06 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087392+0000", 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000313164 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.087425+0000", 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2768999999999997e-05 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187523+0000", 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100097504 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187687+0000", 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000164537 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.187711+0000", 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4209000000000001e-05 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289394+0000", 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10168234399999999 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289476+0000", 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1903999999999994e-05 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.174 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 38/32 ECSubWrite(tid=18, reqid=client.4728.0:194, at_version=38'11, trim_to=0'0, roll_forward_to=36'9))", 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 73.677618819000003, 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202828813, 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:43:02.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487349+0000", 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:43:02.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487637+0000", 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00028784999999999998 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.487673+0000", 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5542e-05 2024-09-16T04:43:02.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.587811+0000", 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100137777 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.688010+0000", 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10019921599999999 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.688025+0000", 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4955e-05 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.690131+0000", 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002106442 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:48.690178+0000", 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6381000000000002e-05 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 44/41 ECSubWrite(tid=55, reqid=osd.4.0:55, at_version=44'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 67.951655700999993, 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.296361333, 2024-09-16T04:43:02.180 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213312+0000", 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2e-07 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213370+0000", 2024-09-16T04:43:02.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7278000000000003e-05 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213391+0000", 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1696999999999999e-05 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.213413+0000", 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1104999999999999e-05 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.407449+0000", 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19403645 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.407462+0000", 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3315e-05 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509615+0000", 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102152944 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.509673+0000", 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8023999999999999e-05 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s2 44/41 ECSubWrite(tid=59, reqid=client.4728.0:299, at_version=44'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 67.941876234000006, 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.28930476599999999, 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223091+0000", 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223094+0000", 2024-09-16T04:43:02.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6359999999999998e-06 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223754+0000", 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00066019600000000005 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223785+0000", 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1165999999999999e-05 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.223806+0000", 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0134999999999999e-05 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.508219+0000", 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.28441328900000001 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.508234+0000", 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5213e-05 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.512334+0000", 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041000660000000003 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:54.512396+0000", 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2064999999999996e-05 2024-09-16T04:43:02.190 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 58/37 ECSubWrite(tid=117, reqid=client.4728.0:455, at_version=58'27, trim_to=0'0, roll_forward_to=57'22))", 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.403858+0000", 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.761109767000001, 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.227776374, 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.191 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403858+0000", 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403858+0000", 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403859+0000", 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.023e-06 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403890+0000", 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1013000000000003e-05 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403917+0000", 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6965999999999999e-05 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.403934+0000", 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7085999999999999e-05 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.629673+0000", 2024-09-16T04:43:02.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.22573882300000001 2024-09-16T04:43:02.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.629696+0000", 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2745999999999999e-05 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.631574+0000", 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001878242 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.631634+0000", 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0475e-05 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 58/53 ECSubWrite(tid=60, reqid=osd.3.0:60, at_version=58'32, trim_to=0'0, roll_forward_to=57'30))", 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.423074+0000", 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.741893752999999, 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20255648300000001, 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.198 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423074+0000", 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423074+0000", 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423076+0000", 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.793e-06 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423130+0000", 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4191999999999999e-05 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423167+0000", 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6770000000000002e-05 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423190+0000", 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3629e-05 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.523695+0000", 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100505223 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.523724+0000", 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8276000000000002e-05 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.625547+0000", 2024-09-16T04:43:02.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101823598 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.625630+0000", 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3002000000000003e-05 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 58/53 ECSubWrite(tid=61, reqid=osd.3.0:61, at_version=58'34, trim_to=0'0, roll_forward_to=57'30))", 2024-09-16T04:43:02.203 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:08.423858+0000", 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.741109452000003, 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20264642399999999, 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423858+0000", 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423858+0000", 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423860+0000", 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.747e-06 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423929+0000", 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9251999999999994e-05 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423965+0000", 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5478999999999997e-05 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.423988+0000", 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3048999999999999e-05 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.624695+0000", 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20070756300000001 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.624728+0000", 2024-09-16T04:43:02.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2353e-05 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.626430+0000", 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001702606 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:08.626505+0000", 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4375000000000005e-05 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.208 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 70/67 ECSubWrite(tid=82, reqid=osd.2.0:82, at_version=70'56, trim_to=0'0, roll_forward_to=69'54))", 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:20.362623+0000", 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 41.802344241999997, 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20271673900000001, 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362623+0000", 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362623+0000", 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362624+0000", 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6300000000000005e-07 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362649+0000", 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4700000000000001e-05 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.362659+0000", 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0521e-05 2024-09-16T04:43:02.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.462788+0000", 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100128973 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.463788+0000", 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099988700000000008 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.463802+0000", 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4321999999999999e-05 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.565297+0000", 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101494468 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:20.565340+0000", 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3305e-05 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 80/67 ECSubWrite(tid=116, reqid=osd.2.0:114, at_version=0'0, trim_to=0'0, roll_forward_to=80'75))", 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.575809+0000", 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.58915897, 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20150364200000001, 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.214 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575809+0000", 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575809+0000", 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575811+0000", 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8640000000000001e-06 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575833+0000", 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2599000000000002e-05 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.575863+0000", 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9484000000000001e-05 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.675964+0000", 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100101125 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676136+0000", 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017185099999999999 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676166+0000", 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0573e-05 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.777230+0000", 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101063783 2024-09-16T04:43:02.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.777312+0000", 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2362999999999997e-05 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 80/53 ECSubWrite(tid=130, reqid=osd.3.0:130, at_version=80'69, trim_to=0'0, roll_forward_to=79'67))", 2024-09-16T04:43:02.219 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.676168+0000", 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.488799663999998, 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20210377600000001, 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676168+0000", 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676168+0000", 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676169+0000", 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0139999999999999e-06 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676206+0000", 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6566000000000002e-05 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.676227+0000", 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1140000000000001e-05 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776333+0000", 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100106442 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876610+0000", 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10027724 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876627+0000", 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6416e-05 2024-09-16T04:43:02.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.878207+0000", 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015805719999999999 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.878272+0000", 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4386000000000003e-05 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.224 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.as3 80/67 ECSubWrite(tid=131, reqid=osd.3.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=80'51))", 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.776388+0000", 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.388579626999999, 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20107434499999999, 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.225 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776388+0000", 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776388+0000", 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776389+0000", 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0640000000000001e-06 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776406+0000", 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7309999999999999e-05 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776425+0000", 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8790000000000001e-05 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.776437+0000", 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2283000000000001e-05 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876715+0000", 2024-09-16T04:43:02.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10027707299999999 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.876731+0000", 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6460000000000002e-05 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.977340+0000", 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100609109 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.977462+0000", 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000122256 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.14s1 80/53 ECSubWrite(tid=132, reqid=osd.3.0:130, at_version=0'0, trim_to=0'0, roll_forward_to=80'69))", 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:30.907366+0000", 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.257602093999999, 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201024537, 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907366+0000", 2024-09-16T04:43:02.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907366+0000", 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907367+0000", 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.933e-06 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907393+0000", 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.584e-05 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907433+0000", 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9403000000000002e-05 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:30.907455+0000", 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2643e-05 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.007692+0000", 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10023647400000001 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.007708+0000", 2024-09-16T04:43:02.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6133e-05 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.108290+0000", 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100581795 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:31.108390+0000", 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010031600000000001 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.235 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es1 90/67 ECSubWrite(tid=146, reqid=osd.2.0:144, at_version=0'0, trim_to=0'0, roll_forward_to=90'84))", 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:41.611525+0000", 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 20.553442314000002, 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20182486499999999, 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611525+0000", 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611525+0000", 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611526+0000", 2024-09-16T04:43:02.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8599999999999999e-07 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611540+0000", 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4209999999999999e-05 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.611558+0000", 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8383e-05 2024-09-16T04:43:02.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.711659+0000", 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100100222 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.811951+0000", 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10029222 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.811984+0000", 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3367000000000001e-05 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.813285+0000", 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013007680000000001 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:41.813350+0000", 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5308999999999994e-05 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 92/67 ECSubWrite(tid=203, reqid=client.4728.0:827, at_version=92'51, trim_to=0'0, roll_forward_to=90'49))", 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:42.724950+0000", 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 19.440017218000001, 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20312651500000001, 2024-09-16T04:43:02.241 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.724950+0000", 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.724950+0000", 2024-09-16T04:43:02.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.724952+0000", 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9460000000000001e-06 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725005+0000", 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2482999999999999e-05 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725041+0000", 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5743e-05 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725064+0000", 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3796999999999999e-05 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.825306+0000", 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100241371 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.825333+0000", 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7716999999999999e-05 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.927997+0000", 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10266338799999999 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.928077+0000", 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0069999999999997e-05 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 92/67 ECSubWrite(tid=204, reqid=osd.4.0:204, at_version=92'53, trim_to=0'0, roll_forward_to=90'49))", 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 19.439852052999999, 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20378613500000001, 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:43:02.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725116+0000", 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9999999999999997e-07 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725162+0000", 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5952999999999999e-05 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.725179+0000", 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7073000000000001e-05 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.825284+0000", 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100105179 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.927398+0000", 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10211313800000001 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.927415+0000", 2024-09-16T04:43:02.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7166000000000001e-05 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.928821+0000", 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001406247 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:42.928902+0000", 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0679000000000005e-05 2024-09-16T04:43:02.251 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPush(3.19s3 95/58 [PushOp(3:9de467d5:::smithi16530038-27:7, version: 50'27, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:9de467d5:::smithi16530038-27:7@50'27, size: 1294336, copy_subset: [], clone_subset: {}, snapset: 11=[]:{4: [3], 7: [7], 11: [11, d, b]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:46.634171+0000", 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.530796389000001, 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20206529000000001, 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.252 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634171+0000", 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634171+0000", 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634173+0000", 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.193e-06 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634920+0000", 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00074700600000000004 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.634992+0000", 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1847000000000005e-05 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.735128+0000", 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100136155 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.835418+0000", 2024-09-16T04:43:02.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100289403 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:46.836237+0000", 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000818686 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.256 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.1s2 97/67 ECSubWrite(tid=216, reqid=osd.4.0:216, at_version=97'57, trim_to=0'0, roll_forward_to=93'55))", 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:47.954069+0000", 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 14.210898630000001, 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20286156699999999, 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954069+0000", 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.257 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954069+0000", 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954071+0000", 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.843e-06 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954112+0000", 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0893000000000003e-05 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:47.954138+0000", 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6432999999999999e-05 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.054245+0000", 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100106473 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.154543+0000", 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10029845499999999 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.154559+0000", 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6276000000000001e-05 2024-09-16T04:43:02.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.156854+0000", 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022942919999999999 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:48.156931+0000", 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6902000000000003e-05 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.261 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPushReply(3.12s0 103/102 [PushReplyOp(3:4cc0f814:::smithi16530038-19:head)])", 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:54.612178+0000", 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 7.5527898210000002, 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20150238500000001, 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.612178+0000", 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.612178+0000", 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.612178+0000", 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3200000000000001e-07 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.612181+0000", 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1839999999999999e-06 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.612187+0000", 2024-09-16T04:43:02.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9529999999999996e-06 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.612195+0000", 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3459999999999996e-06 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.812788+0000", 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20059300899999999 2024-09-16T04:43:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:54.813680+0000", 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000892661 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.11s3 104/102 ECSubWrite(tid=44, reqid=client.4728.0:969, at_version=104'54, trim_to=0'0, roll_forward_to=103'50))", 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:55.319438+0000", 2024-09-16T04:43:02.266 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.8455300690000005, 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30509491, 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.319438+0000", 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.267 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.319438+0000", 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.319439+0000", 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6959999999999999e-06 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.319485+0000", 2024-09-16T04:43:02.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5277e-05 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.319514+0000", 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9946000000000001e-05 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.419674+0000", 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100159466 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.620086+0000", 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20041244999999999 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.620107+0000", 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0443000000000001e-05 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2024-09-16T04:43:02.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.624372+0000", 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042654930000000004 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:55.624532+0000", 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016013900000000001 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.271 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDPGPush(3.3s3 107/103 [PushOp(3:c31e3ec6:::smithi16530038-50:19, version: 97'42, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:c31e3ec6:::smithi16530038-50:19@97'42, size: 1818624, copy_subset: [], clone_subset: {}, snapset: 19=[]:{19: [19]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:42:57.836285+0000", 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3286823459999999, 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20232013600000001, 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.272 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:57.836285+0000", 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:57.836285+0000", 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:57.836287+0000", 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:43:02.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:57.836643+0000", 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00035560000000000002 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:57.836689+0000", 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.634e-05 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:57.936832+0000", 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10014263900000001 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:58.037238+0000", 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10040572 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:42:58.038605+0000", 2024-09-16T04:43:02.276 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013677470000000001 2024-09-16T04:43:02.276 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.276 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.276 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.276 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.276 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.276 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:02.276 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:43:02.278 INFO:tasks.rados.rados.0.smithi165.stdout:398: done (0 left) 2024-09-16T04:43:02.279 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:43:02.280 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:02.270+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:02.280 INFO:tasks.rados.rados.0.smithi165.stderr:0 errors. 2024-09-16T04:43:02.280 INFO:tasks.rados.rados.0.smithi165.stderr: 2024-09-16T04:43:02.281 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:43:02.378 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:02.374+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:02.458 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:02.458 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:43:02.458 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2024-09-16T04:43:02.458 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:02.473 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:43:02.479 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:02.475+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:02.580 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:02.574+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:02.636 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:02.637 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2024-09-16T04:43:02.637 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2024-09-16T04:43:02.637 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2024-09-16T04:43:02.637 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:02.637 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:02.638 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":111,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:43:01.143271+0000","last_up_change":"2024-09-16T04:42:51.561897+0000","last_in_change":"2024-09-16T04:42:52.575951+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:41:23.074365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:41:25.522114+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:41:39.714918+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":35,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":35,"last_pg_merge_meta":{"source_pgid":"3.23","ready_epoch":110,"last_epoch_started":104,"last_epoch_clean":109,"source_version":"60'17","target_version":"97'43"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"110","last_force_op_resend_preluminous":"103","auid":0,"snap_mode":"selfmanaged","snap_seq":37,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6801","nonce":1228226348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6803","nonce":1228226348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6807","nonce":1228226348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6805","nonce":1228226348}]},"public_addr":"172.21.15.59:6801/1228226348","cluster_addr":"172.21.15.59:6803/1228226348","heartbeat_back_addr":"172.21.15.59:6807/1228226348","heartbeat_front_addr":"172.21.15.59:6805/1228226348","state":["exists","up"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6817","nonce":2731115256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6819","nonce":2731115256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6823","nonce":2731115256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6821","nonce":2731115256}]},"public_addr":"172.21.15.139:6817/2731115256","cluster_addr":"172.21.15.139:6819/2731115256","heartbeat_back_addr":"172.21.15.139:6823/2731115256","heartbeat_front_addr":"172.21.15.139:6821/2731115256","state":["exists","up"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6825","nonce":76637888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6827","nonce":76637888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6831","nonce":76637888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6829","nonce":76637888}]},"public_addr":"172.21.15.144:6825/76637888","cluster_addr":"172.21.15.144:6827/76637888","heartbeat_back_addr":"172.21.15.144:6831/76637888","heartbeat_front_addr":"172.21.15.144:6829/76637888","state":["exists","up"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6803","nonce":58455506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6805","nonce":58455506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6809","nonce":58455506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6807","nonce":58455506}]},"public_addr":"172.21.15.165:6803/58455506","cluster_addr":"172.21.15.165:6805/58455506","heartbeat_back_addr":"172.21.15.165:6809/58455506","heartbeat_front_addr":"172.21.15.165:6807/58455506","state":["exists","up"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6825","nonce":2712399685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6827","nonce":2712399685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6831","nonce":2712399685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6829","nonce":2712399685}]},"public_addr":"172.21.15.59:6825/2712399685","cluster_addr":"172.21.15.59:6827/2712399685","heartbeat_back_addr":"172.21.15.59:6831/2712399685","heartbeat_front_addr":"172.21.15.59:6829/2712399685","state":["exists","up"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6801","nonce":2946170755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6803","nonce":2946170755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6807","nonce":2946170755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6805","nonce":2946170755}]},"public_addr":"172.21.15.139:6801/2946170755","cluster_addr":"172.21.15.139:6803/2946170755","heartbeat_back_addr":"172.21.15.139:6807/2946170755","heartbeat_front_addr":"172.21.15.139:6805/2946170755","state":["exists","up"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6801","nonce":3812816248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6803","nonce":3812816248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6807","nonce":3812816248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6805","nonce":3812816248}]},"public_addr":"172.21.15.144:6801/3812816248","cluster_addr":"172.21.15.144:6803/3812816248","heartbeat_back_addr":"172.21.15.144:6807/3812816248","heartbeat_front_addr":"172.21.15.144:6805/3812816248","state":["exists","up"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6819","nonce":1352862134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6821","nonce":1352862134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6825","nonce":1352862134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6823","nonce":1352862134}]},"public_addr":"172.21.15.165:6819/1352862134","cluster_addr":"172.21.15.165:6821/1352862134","heartbeat_back_addr":"172.21.15.165:6825/1352862134","heartbeat_front_addr":"172.21.15.165:6823/1352862134","state":["exists","up"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6817","nonce":1451673997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6819","nonce":1451673997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6823","nonce":1451673997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6821","nonce":1451673997}]},"public_addr":"172.21.15.59:6817/1451673997","cluster_addr":"172.21.15.59:6819/1451673997","heartbeat_back_addr":"172.21.15.59:6823/1451673997","heartbeat_front_addr":"172.21.15.59:6821/1451673997","state":["exists","up"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6809","nonce":174106571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6811","nonce":174106571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6815","nonce":174106571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6813","nonce":174106571}]},"public_addr":"172.21.15.139:6809/174106571","cluster_addr":"172.21.15.139:6811/174106571","heartbeat_back_addr":"172.21.15.139:6815/174106571","heartbeat_front_addr":"172.21.15.139:6813/174106571","state":["exists","up"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6809","nonce":66779937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6812","nonce":66779937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6818","nonce":66779937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6815","nonce":66779937}]},"public_addr":"172.21.15.144:6809/66779937","cluster_addr":"172.21.15.144:6812/66779937","heartbeat_back_addr":"172.21.15.144:6818/66779937","heartbeat_front_addr":"172.21.15.144:6815/66779937","state":["exists","up"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6811","nonce":49204916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6813","nonce":49204916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6817","nonce":49204916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6815","nonce":49204916}]},"public_addr":"172.21.15.165:6811/49204916","cluster_addr":"172.21.15.165:6813/49204916","heartbeat_back_addr":"172.21.15.165:6817/49204916","heartbeat_front_addr":"172.21.15.165:6815/49204916","state":["exists","up"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6809","nonce":3097346146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6811","nonce":3097346146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6815","nonce":3097346146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6813","nonce":3097346146}]},"public_addr":"172.21.15.59:6809/3097346146","cluster_addr":"172.21.15.59:6811/3097346146","heartbeat_back_addr":"172.21.15.59:6815/3097346146","heartbeat_front_addr":"172.21.15.59:6813/3097346146","state":["exists","up"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6825","nonce":2903817823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6827","nonce":2903817823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6831","nonce":2903817823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6829","nonce":2903817823}]},"public_addr":"172.21.15.139:6825/2903817823","cluster_addr":"172.21.15.139:6827/2903817823","heartbeat_back_addr":"172.21.15.139:6831/2903817823","heartbeat_front_addr":"172.21.15.139:6829/2903817823","state":["exists","up"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6811","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6814","nonce":229023981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6819","nonce":229023981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6823","nonce":229023981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6821","nonce":229023981}]},"public_addr":"172.21.15.144:6814/229023981","cluster_addr":"172.21.15.144:6819/229023981","heartbeat_back_addr":"172.21.15.144:6823/229023981","heartbeat_front_addr":"172.21.15.144:6821/229023981","state":["exists","up"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":53,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6827","nonce":2758676961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6829","nonce":2758676961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6833","nonce":2758676961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":2758676961},{"type":"v1","addr":"172.21.15.165:6831","nonce":2758676961}]},"public_addr":"172.21.15.165:6827/2758676961","cluster_addr":"172.21.15.165:6829/2758676961","heartbeat_back_addr":"172.21.15.165:6833/2758676961","heartbeat_front_addr":"172.21.15.165:6831/2758676961","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:42.317646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:31.645001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:36.495271+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:48.717655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:32.926273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:47.648804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:49.832912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:43.126550+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:38.541154+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:30.085848+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:36.730415+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:30.702050+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:48.659730+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:51.158730+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:52.799239+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T04:41:21.867981+0000","dead_epoch":101}],"pg_upmap":[{"pgid":"3.10","osds":[2,7,1,14,5,3,15]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.19","osds":[14,1,0,3,15,8,10]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3,"length":1},{"begin":7,"length":1},{"begin":9,"length":16},{"begin":26,"length":1},{"begin":28,"length":1},{"begin":35,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:43:02.650 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:43:02.653 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:43:02.657 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:02.681 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:43:02.675+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:43:02.782 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:43:02.779+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:43:02.816 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-16T04:43:02.816 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2024-09-16T04:43:02.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2024-09-16T04:43:02.816 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 98.437511753999999, 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1755266710000001, 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 1 2024-09-16T04:43:02.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374329+0000", 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374332+0000", 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8059999999999999e-06 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374345+0000", 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2529e-05 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374365+0000", 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0695000000000001e-05 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374383+0000", 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7569000000000001e-05 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374514+0000", 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013070099999999999 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:24.374544+0000", 2024-09-16T04:43:02.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0731000000000001e-05 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.343896+0000", 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.969351224 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344169+0000", 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027390100000000001 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.821 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.344583+0000", 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00041308600000000001 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.346047+0000", 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014642699999999999 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549695+0000", 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.203648463 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549791+0000", 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6137999999999993e-05 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:26.549856+0000", 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4558000000000001e-05 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.191095575000006, 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51346886800000002, 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 111 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.824 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620746+0000", 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620747+0000", 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4959999999999999e-06 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620757+0000", 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0316000000000001e-05 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620778+0000", 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0613000000000001e-05 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620792+0000", 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4362000000000001e-05 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928510+0000", 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.307717717 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928629+0000", 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011919 2024-09-16T04:43:02.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.928822+0000", 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019246199999999999 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930635+0000", 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00181292 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134065+0000", 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20343018399999999 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134149+0000", 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4635000000000005e-05 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134214+0000", 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4973000000000003e-05 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.829 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.191025212, 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51366009000000001, 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.830 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 112 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620816+0000", 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.620817+0000", 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3260000000000001e-06 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621035+0000", 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00021731100000000001 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621061+0000", 2024-09-16T04:43:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6744999999999999e-05 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621082+0000", 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1061999999999999e-05 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929460+0000", 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.308377346 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929595+0000", 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013512600000000001 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.929798+0000", 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020277199999999999 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932465+0000", 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002667862 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134319+0000", 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20185381599999999 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134407+0000", 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7894999999999998e-05 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134476+0000", 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8829000000000001e-05 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.190725268999998, 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51357863199999998, 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 113 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621116+0000", 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621118+0000", 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.689e-06 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621129+0000", 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1049999999999999e-05 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621164+0000", 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5367999999999997e-05 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621182+0000", 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8352e-05 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930689+0000", 2024-09-16T04:43:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.309507015 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.930800+0000", 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000110871 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931026+0000", 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000226293 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.933569+0000", 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002542786 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134566+0000", 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.200996272 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134640+0000", 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4412000000000005e-05 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134694+0000", 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4524000000000002e-05 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.190628521999997, 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51372108500000002, 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 114 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621213+0000", 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621214+0000", 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4300000000000001e-06 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621225+0000", 2024-09-16T04:43:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1348999999999999e-05 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621245+0000", 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9966000000000002e-05 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621259+0000", 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4102e-05 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931642+0000", 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31038274300000002 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931776+0000", 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133552 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.931973+0000", 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000197262 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.033893+0000", 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101919751 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134774+0000", 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.10088169600000001 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134847+0000", 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2309000000000002e-05 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134934+0000", 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6924999999999999e-05 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.190556713000007, 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51387291300000004, 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 115 2024-09-16T04:43:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621284+0000", 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621286+0000", 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.26e-06 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621492+0000", 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020643500000000001 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621517+0000", 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4989000000000001e-05 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621534+0000", 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6515999999999999e-05 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932515+0000", 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.31098146799999998 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932631+0000", 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011636 2024-09-16T04:43:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.932814+0000", 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000182406 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.134236+0000", 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.201422618 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135029+0000", 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079253199999999998 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135102+0000", 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3286000000000003e-05 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.135157+0000", 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5043000000000001e-05 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.190274895000002, 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.717172162, 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 116 2024-09-16T04:43:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621566+0000", 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621568+0000", 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621606+0000", 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8167999999999998e-05 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621628+0000", 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1742e-05 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621643+0000", 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5696e-05 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235313+0000", 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61366968300000002 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235449+0000", 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136036 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.235657+0000", 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207405 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237049+0000", 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001392552 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338582+0000", 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.101532626 2024-09-16T04:43:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338670+0000", 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7888999999999998e-05 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.338738+0000", 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8769000000000005e-05 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 90.190166904999998, 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.81671015999999996, 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 117 2024-09-16T04:43:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621674+0000", 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621676+0000", 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4440000000000001e-06 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621686+0000", 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0845e-05 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621706+0000", 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9168999999999999e-05 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:32.621723+0000", 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7538999999999999e-05 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236241+0000", 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.61451815600000004 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236372+0000", 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013062300000000001 2024-09-16T04:43:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.236574+0000", 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020165600000000001 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.237570+0000", 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099642999999999997 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438224+0000", 2024-09-16T04:43:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20065369 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438317+0000", 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3624999999999999e-05 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:33.438384+0000", 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6983000000000006e-05 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 88.517092927999997, 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.508075677, 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 138 2024-09-16T04:43:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294748+0000", 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294750+0000", 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294760+0000", 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294779+0000", 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8941999999999999e-05 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294794+0000", 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5038999999999999e-05 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405306+0000", 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11051159300000001 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405426+0000", 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011977 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.405611+0000", 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000185217 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709561+0000", 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30394976699999998 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802549+0000", 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.092988137999999998 2024-09-16T04:43:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802707+0000", 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015861799999999999 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.802824+0000", 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116517 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 88.517022804999996, 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60756662299999997, 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 139 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294818+0000", 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294820+0000", 2024-09-16T04:43:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4330000000000001e-06 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294830+0000", 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0023e-05 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294850+0000", 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0622e-05 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294864+0000", 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4042e-05 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406110+0000", 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.11124526 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406222+0000", 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111871 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.406398+0000", 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017598000000000001 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.709577+0000", 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.30317971500000002 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902101+0000", 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.19252423499999999 2024-09-16T04:43:02.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902267+0000", 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016537499999999999 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902385+0000", 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011806700000000001 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 88.516930677000005, 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.60781196900000001, 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 140 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294910+0000", 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.294912+0000", 2024-09-16T04:43:02.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.407e-06 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295040+0000", 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012836600000000001 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295062+0000", 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1682e-05 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295083+0000", 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1109000000000001e-05 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507008+0000", 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.21192486699999999 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507149+0000", 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000141042 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.507369+0000", 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022046500000000001 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710298+0000", 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.202928308 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902525+0000", 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.192226958 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902611+0000", 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6274999999999996e-05 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.902722+0000", 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011149 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.884 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 88.516703096000001, 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.70811914799999998, 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 141 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295138+0000", 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295140+0000", 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5200000000000001e-06 2024-09-16T04:43:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295152+0000", 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2053e-05 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295172+0000", 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0282e-05 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.295191+0000", 2024-09-16T04:43:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8759000000000001e-05 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508486+0000", 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.213295545 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508630+0000", 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014419899999999999 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.508824+0000", 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019358600000000001 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:34.710318+0000", 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20149381 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.002981+0000", 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.29266302500000002 2024-09-16T04:43:02.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003144+0000", 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016330500000000001 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.003257+0000", 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113064 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.890 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.852246367999996, 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51415646199999998, 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 159 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959595+0000", 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959597+0000", 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9190000000000002e-06 2024-09-16T04:43:02.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959609+0000", 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2797e-05 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959629+0000", 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0035e-05 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959645+0000", 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5786000000000001e-05 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167529+0000", 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.207883768 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.167762+0000", 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023295499999999999 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.168134+0000", 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037181099999999999 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.270830+0000", 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.102696683 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473585+0000", 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20275470700000001 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473677+0000", 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1772000000000002e-05 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473751+0000", 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4228999999999997e-05 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.852165606, 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51420228400000001, 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 160 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959675+0000", 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959677+0000", 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3769999999999999e-06 2024-09-16T04:43:02.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959687+0000", 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959711+0000", 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3811999999999998e-05 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959727+0000", 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5446000000000001e-05 2024-09-16T04:43:02.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169193+0000", 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.209466241 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169340+0000", 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000147279 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.169545+0000", 2024-09-16T04:43:02.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000205167 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374117+0000", 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20457175999999999 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473763+0000", 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099646306000000004 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473828+0000", 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4046000000000003e-05 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473878+0000", 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0241000000000002e-05 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:43:02.902 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.852088327000004, 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51424688600000001, 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 161 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959753+0000", 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959754+0000", 2024-09-16T04:43:02.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959920+0000", 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166088 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959944+0000", 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.959963+0000", 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9176000000000001e-05 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371049+0000", 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41108644500000002 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371185+0000", 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013619899999999999 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.371395+0000", 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020912 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.377708+0000", 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.006313173 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473891+0000", 2024-09-16T04:43:02.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.096182904 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473949+0000", 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8016999999999997e-05 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474000+0000", 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0995000000000002e-05 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.908 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.851772569999994, 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51552797100000003, 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 162 2024-09-16T04:43:02.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960069+0000", 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960070+0000", 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1930000000000001e-06 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960077+0000", 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6830000000000008e-06 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960091+0000", 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3653e-05 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960101+0000", 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7629999999999995e-06 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372554+0000", 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41245356599999999 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.372785+0000", 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023023100000000001 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.373156+0000", 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000371601 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474010+0000", 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.100854291 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475328+0000", 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001317304 2024-09-16T04:43:02.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475477+0000", 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014954899999999999 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.475596+0000", 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011913699999999999 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.851724341999997, 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51788103100000005, 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 163 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.915 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960117+0000", 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960118+0000", 2024-09-16T04:43:02.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6000000000000002e-07 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960124+0000", 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9460000000000003e-06 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960134+0000", 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0115999999999999e-05 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960144+0000", 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374237+0000", 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41409377600000002 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374462+0000", 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00022478600000000001 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.374797+0000", 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000334586 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474018+0000", 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.099220947000000004 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477813+0000", 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003795171 2024-09-16T04:43:02.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477921+0000", 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010822100000000001 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.477998+0000", 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6662999999999994e-05 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.851685128, 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51822496799999995, 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 164 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960156+0000", 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960157+0000", 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7399999999999991e-07 2024-09-16T04:43:02.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960241+0000", 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3677999999999994e-05 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960256+0000", 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5206e-05 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960267+0000", 2024-09-16T04:43:02.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1372999999999999e-05 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375650+0000", 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.41538323100000002 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.375885+0000", 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000234758 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.376293+0000", 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00040777599999999999 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474024+0000", 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.097731440000000003 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478113+0000", 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004088225 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478261+0000", 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014825 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478381+0000", 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012005700000000001 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4201.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 86.851559054999996, 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51862660500000002, 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4201", 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/1070274904", 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 165 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960282+0000", 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960283+0000", 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9000000000000005e-07 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960289+0000", 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.49e-06 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960301+0000", 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2585999999999999e-05 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:35.960309+0000", 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9629999999999998e-06 2024-09-16T04:43:02.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472635+0000", 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.51232596699999999 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.472771+0000", 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000136243 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for subops from 2", 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.473033+0000", 2024-09-16T04:43:02.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026155699999999999 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op_commit", 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.474539+0000", 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015065 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478656+0000", 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041164319999999997 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2024-09-16T04:43:02.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478793+0000", 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013757299999999999 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:36.478909+0000", 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011530400000000001 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4728.0:153 3.3s0 3:c31e3ec6:::smithi16530038-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:02.932 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 76.522259923000007, 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.511692229, 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "continuous": false, 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4728", 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 153 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289581+0000", 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289583+0000", 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.04e-06 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289590+0000", 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4899999999999997e-06 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289610+0000", 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9814999999999999e-05 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.289625+0000", 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5553000000000002e-05 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.296844+0000", 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0072185369999999997 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.297045+0000", 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020125199999999999 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599740+0000", 2024-09-16T04:43:02.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.302695037 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.599866+0000", 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012644500000000001 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2024-09-16T04:41:46.801273+0000", 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.20140706 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2024-09-16T04:43:02.938 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-16T04:43:02.938 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-16T04:43:02.939 INFO:teuthology.orchestra.run.smithi165.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-09-16T04:43:02.940 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:43:02.940 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.15 2024-09-16T04:43:02.940 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-16T04:43:02.941 INFO:tasks.thrashosds.thrasher:applying new sharding: 2024-09-16T04:43:02.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-15 --sharding '' reshard 2024-09-16T04:43:02.941 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:43:02.934+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:43:02.955 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:02.956 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:43:02.971 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-16T04:43:02.654649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119308851,"num_objects":99,"num_object_clones":48,"num_object_copies":673,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":1,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":7,"num_read":322,"num_read_kb":96374,"num_write":531,"num_write_kb":72062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":77922304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1957,"ondisk_log_size":1957,"up":263,"acting":263,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":290,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":748252,"kb_used_data":251264,"kb_used_omap":23,"kb_used_meta":496616,"kb_avail":1414829348,"statfs":{"total":1449551462400,"available":1448785252352,"internally_reserved":0,"allocated":257294336,"data_stored":318287258,"data_compressed":3941583,"data_compressed_allocated":5390336,"data_compressed_original":11640832,"omap_allocated":23928,"internal_metadata":508535432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2408448,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":-19,"num_objects_missing":-19,"num_objects_degraded":-113,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":10,"num_read_kb":675,"num_write":29,"num_write_kb":2352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":38584320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027551"},"pg_stats":[{"pgid":"3.22","version":"95'33","reported_seq":69,"reported_epoch":110,"state":"active+recovering","last_fresh":"2024-09-16T04:43:00.562788+0000","last_change":"2024-09-16T04:42:57.652205+0000","last_active":"2024-09-16T04:43:00.562788+0000","last_peered":"2024-09-16T04:43:00.562788+0000","last_clean":"2024-09-16T04:42:45.755640+0000","last_became_active":"2024-09-16T04:42:55.111975+0000","last_became_peered":"2024-09-16T04:42:55.111975+0000","last_unstale":"2024-09-16T04:43:00.562788+0000","last_undegraded":"2024-09-16T04:43:00.562788+0000","last_fullsized":"2024-09-16T04:43:00.562788+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":6,"last_scrub":"72'17","last_scrub_stamp":"2024-09-16T04:42:26.786330+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.786330+0000","objects_scrubbed":1,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":843,"objects_trimmed":0,"snaptrim_duration":0.78087386000000003,"stat_sum":{"num_bytes":6373376,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":6373376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,12,1,11],"acting":[13,2,3,6,12,1,11],"avail_no_missing":["0(2)","2(0)","4(4)","5(1)","7(5)","11(3)","14(6)"],"object_location_counts":[{"shards":"0(2),2(0),4(4),5(1),7(5),11(3),14(6)","objects":2},{"shards":"1(5),2(1),3(2),6(3),11(6),12(4),13(0)","objects":1}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"f"},{"start":"1c","length":"1"}]},{"pgid":"3.21","version":"105'49","reported_seq":97,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.656735+0000","last_change":"2024-09-16T04:42:57.726707+0000","last_active":"2024-09-16T04:42:59.656735+0000","last_peered":"2024-09-16T04:42:59.656735+0000","last_clean":"2024-09-16T04:42:59.656735+0000","last_became_active":"2024-09-16T04:42:49.166894+0000","last_became_peered":"2024-09-16T04:42:49.166894+0000","last_unstale":"2024-09-16T04:42:59.656735+0000","last_undegraded":"2024-09-16T04:42:59.656735+0000","last_fullsized":"2024-09-16T04:42:59.656735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'31","last_scrub_stamp":"2024-09-16T04:42:26.688435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.688435+0000","objects_scrubbed":1,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:27.753309+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0.103134376,"stat_sum":{"num_bytes":2375680,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.20","version":"36'4","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.649794+0000","last_change":"2024-09-16T04:42:57.640313+0000","last_active":"2024-09-16T04:42:59.649794+0000","last_peered":"2024-09-16T04:42:59.649794+0000","last_clean":"2024-09-16T04:42:59.649794+0000","last_became_active":"2024-09-16T04:42:53.909464+0000","last_became_peered":"2024-09-16T04:42:53.909464+0000","last_unstale":"2024-09-16T04:42:59.649794+0000","last_undegraded":"2024-09-16T04:42:59.649794+0000","last_fullsized":"2024-09-16T04:42:59.649794+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.091321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.091321+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:03.319640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025490228,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11,1,5,6,4],"acting":[14,2,11,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1f","version":"105'64","reported_seq":89,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T04:42:57.727069+0000","last_change":"2024-09-16T04:42:57.828210+0000","last_active":"2024-09-16T04:42:57.727069+0000","last_peered":"2024-09-16T04:42:57.727069+0000","last_clean":"2024-09-16T04:42:57.727069+0000","last_became_active":"2024-09-16T04:42:48.879345+0000","last_became_peered":"2024-09-16T04:42:48.879345+0000","last_unstale":"2024-09-16T04:42:57.727069+0000","last_undegraded":"2024-09-16T04:42:57.727069+0000","last_fullsized":"2024-09-16T04:42:57.727069+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'52","last_scrub_stamp":"2024-09-16T04:42:25.895250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.895250+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:35.355778+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0.102254495,"stat_sum":{"num_bytes":1867776,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":10947,"num_write":5,"num_write_kb":1824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1867776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1e","version":"107'106","reported_seq":154,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.784074+0000","last_change":"2024-09-16T04:42:57.832704+0000","last_active":"2024-09-16T04:42:59.784074+0000","last_peered":"2024-09-16T04:42:59.784074+0000","last_clean":"2024-09-16T04:42:59.784074+0000","last_became_active":"2024-09-16T04:42:49.077358+0000","last_became_peered":"2024-09-16T04:42:49.077358+0000","last_unstale":"2024-09-16T04:42:59.784074+0000","last_undegraded":"2024-09-16T04:42:59.784074+0000","last_fullsized":"2024-09-16T04:42:59.784074+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"74'53","last_scrub_stamp":"2024-09-16T04:42:27.214170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.214170+0000","objects_scrubbed":5,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:43.567773+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0.109775179,"stat_sum":{"num_bytes":7618560,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":8932,"num_write":15,"num_write_kb":4464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6963200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1d","version":"108'169","reported_seq":183,"reported_epoch":109,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T04:42:59.851696+0000","last_change":"2024-09-16T04:42:59.751593+0000","last_active":"2024-09-16T04:42:59.851696+0000","last_peered":"2024-09-16T04:42:59.851696+0000","last_clean":"2024-09-16T04:42:59.851696+0000","last_became_active":"2024-09-16T04:42:58.938669+0000","last_became_peered":"2024-09-16T04:42:58.938669+0000","last_unstale":"2024-09-16T04:42:59.851696+0000","last_undegraded":"2024-09-16T04:42:59.851696+0000","last_fullsized":"2024-09-16T04:42:59.851696+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'133","last_scrub_stamp":"2024-09-16T04:42:36.357029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:36.357029+0000","objects_scrubbed":6,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:44.900217+0000","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0.50844499099999996,"stat_sum":{"num_bytes":11501568,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5332,"num_write":11,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"3.1c","version":"90'58","reported_seq":482,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T04:42:57.726158+0000","last_change":"2024-09-16T04:42:57.726957+0000","last_active":"2024-09-16T04:42:57.726158+0000","last_peered":"2024-09-16T04:42:57.726158+0000","last_clean":"2024-09-16T04:42:57.726158+0000","last_became_active":"2024-09-16T04:42:48.867904+0000","last_became_peered":"2024-09-16T04:42:48.867904+0000","last_unstale":"2024-09-16T04:42:57.726158+0000","last_undegraded":"2024-09-16T04:42:57.726158+0000","last_fullsized":"2024-09-16T04:42:57.726158+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'54","last_scrub_stamp":"2024-09-16T04:42:25.992723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.992723+0000","objects_scrubbed":2,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:50.476085+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001276606,"stat_sum":{"num_bytes":229376,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":673,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":229376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1b","version":"63'44","reported_seq":62,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.654006+0000","last_change":"2024-09-16T04:42:57.629335+0000","last_active":"2024-09-16T04:42:59.654006+0000","last_peered":"2024-09-16T04:42:59.654006+0000","last_clean":"2024-09-16T04:42:59.654006+0000","last_became_active":"2024-09-16T04:42:48.766598+0000","last_became_peered":"2024-09-16T04:42:48.766598+0000","last_unstale":"2024-09-16T04:42:59.654006+0000","last_undegraded":"2024-09-16T04:42:59.654006+0000","last_fullsized":"2024-09-16T04:42:59.654006+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'44","last_scrub_stamp":"2024-09-16T04:42:25.681915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.681915+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:02.736211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003329668,"stat_sum":{"num_bytes":2990080,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":965,"num_write":20,"num_write_kb":2920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1a","version":"105'69","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.655553+0000","last_change":"2024-09-16T04:42:57.827315+0000","last_active":"2024-09-16T04:42:59.655553+0000","last_peered":"2024-09-16T04:42:59.655553+0000","last_clean":"2024-09-16T04:42:59.655553+0000","last_became_active":"2024-09-16T04:42:49.062656+0000","last_became_peered":"2024-09-16T04:42:49.062656+0000","last_unstale":"2024-09-16T04:42:59.655553+0000","last_undegraded":"2024-09-16T04:42:59.655553+0000","last_fullsized":"2024-09-16T04:42:59.655553+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'51","last_scrub_stamp":"2024-09-16T04:42:35.060495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.060495+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:47.031888+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0.10248621199999999,"stat_sum":{"num_bytes":7405568,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":5360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5914624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.19","version":"102'46","reported_seq":470,"reported_epoch":109,"state":"active+undersized+degraded+remapped","last_fresh":"2024-09-16T04:42:59.650014+0000","last_change":"2024-09-16T04:42:54.104386+0000","last_active":"2024-09-16T04:42:59.650014+0000","last_peered":"2024-09-16T04:42:59.650014+0000","last_clean":"2024-09-16T04:42:06.828625+0000","last_became_active":"2024-09-16T04:42:53.401433+0000","last_became_peered":"2024-09-16T04:42:53.401433+0000","last_unstale":"2024-09-16T04:42:59.650014+0000","last_undegraded":"2024-09-16T04:42:53.508182+0000","last_fullsized":"2024-09-16T04:42:52.588994+0000","mapping_epoch":101,"log_start":"44'22","ondisk_log_start":"44'22","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":22,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.098972921000000005,"stat_sum":{"num_bytes":6340608,"num_objects":10,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":11622,"num_write":29,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":9273344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[14,1,0,3,2147483647,8,10],"avail_no_missing":["14(0)","0(2)","1(1)","3(3)","8(5)","10(6)"],"object_location_counts":[{"shards":"0(2),1(1),3(3),8(5),10(6),14(0)","objects":10}],"blocked_by":[],"up_primary":8,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.18","version":"44'8","reported_seq":428,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T04:43:00.148410+0000","last_change":"2024-09-16T04:42:57.826940+0000","last_active":"2024-09-16T04:43:00.148410+0000","last_peered":"2024-09-16T04:43:00.148410+0000","last_clean":"2024-09-16T04:43:00.148410+0000","last_became_active":"2024-09-16T04:42:12.629523+0000","last_became_peered":"2024-09-16T04:42:12.629523+0000","last_unstale":"2024-09-16T04:43:00.148410+0000","last_undegraded":"2024-09-16T04:43:00.148410+0000","last_fullsized":"2024-09-16T04:43:00.148410+0000","mapping_epoch":61,"log_start":"44'8","ondisk_log_start":"44'8","created":55,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'8","last_scrub_stamp":"2024-09-16T04:42:14.378029+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:14.378029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:40.711392+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0.101625202,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,0,4,5,7,12],"acting":[13,6,0,4,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.17","version":"105'12","reported_seq":307,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.886180+0000","last_change":"2024-09-16T04:42:57.623204+0000","last_active":"2024-09-16T04:42:59.886180+0000","last_peered":"2024-09-16T04:42:59.886180+0000","last_clean":"2024-09-16T04:42:59.886180+0000","last_became_active":"2024-09-16T04:42:04.604172+0000","last_became_peered":"2024-09-16T04:42:04.604172+0000","last_unstale":"2024-09-16T04:42:59.886180+0000","last_undegraded":"2024-09-16T04:42:59.886180+0000","last_fullsized":"2024-09-16T04:42:59.886180+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'6","last_scrub_stamp":"2024-09-16T04:42:11.540212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:11.540212+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:13.374725+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0.0023141920000000001,"stat_sum":{"num_bytes":606208,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":3554,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.16","version":"92'16","reported_seq":301,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T04:42:57.726232+0000","last_change":"2024-09-16T04:42:57.827452+0000","last_active":"2024-09-16T04:42:57.726232+0000","last_peered":"2024-09-16T04:42:57.726232+0000","last_clean":"2024-09-16T04:42:57.726232+0000","last_became_active":"2024-09-16T04:42:53.801354+0000","last_became_peered":"2024-09-16T04:42:53.801354+0000","last_unstale":"2024-09-16T04:42:57.726232+0000","last_undegraded":"2024-09-16T04:42:57.726232+0000","last_fullsized":"2024-09-16T04:42:57.726232+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:57.977558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:57.977558+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:00.719447+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.101716587,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4,8,9,5,10],"acting":[11,1,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.15","version":"36'6","reported_seq":316,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T04:43:01.159622+0000","last_change":"2024-09-16T04:42:57.720101+0000","last_active":"2024-09-16T04:43:01.159622+0000","last_peered":"2024-09-16T04:43:01.159622+0000","last_clean":"2024-09-16T04:43:01.159622+0000","last_became_active":"2024-09-16T04:42:54.005049+0000","last_became_peered":"2024-09-16T04:42:54.005049+0000","last_unstale":"2024-09-16T04:43:01.159622+0000","last_undegraded":"2024-09-16T04:43:01.159622+0000","last_fullsized":"2024-09-16T04:43:01.159622+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'6","last_scrub_stamp":"2024-09-16T04:42:10.772708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.772708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:18.750984+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.0022826220000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,1],"acting":[9,6,3,13,14,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.14","version":"105'103","reported_seq":413,"reported_epoch":109,"state":"active+undersized+remapped","last_fresh":"2024-09-16T04:42:59.890627+0000","last_change":"2024-09-16T04:42:59.991707+0000","last_active":"2024-09-16T04:42:59.890627+0000","last_peered":"2024-09-16T04:42:59.890627+0000","last_clean":"2024-09-16T04:42:50.903791+0000","last_became_active":"2024-09-16T04:42:54.894692+0000","last_became_peered":"2024-09-16T04:42:54.894692+0000","last_unstale":"2024-09-16T04:42:59.890627+0000","last_undegraded":"2024-09-16T04:42:54.601829+0000","last_fullsized":"2024-09-16T04:42:54.597343+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":899,"objects_trimmed":0,"snaptrim_duration":0.402922314,"stat_sum":{"num_bytes":8716288,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":9,"num_read_kb":5332,"num_write":41,"num_write_kb":7280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":14368768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8,1,13,5,4],"acting":[3,9,2147483647,1,13,2147483647,4],"avail_no_missing":["3(0)","1(3)","4(6)","5(5)","8(2)","8(5)","9(1)","13(4)"],"object_location_counts":[{"shards":"1(3),3(0),4(6),9(1),13(4)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1c","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T04:43:00.150704+0000","last_change":"2024-09-16T04:42:53.671490+0000","last_active":"2024-09-16T04:43:00.150704+0000","last_peered":"2024-09-16T04:43:00.150704+0000","last_clean":"2024-09-16T04:43:00.150704+0000","last_became_active":"2024-09-16T04:42:53.671176+0000","last_became_peered":"2024-09-16T04:42:53.671176+0000","last_unstale":"2024-09-16T04:43:00.150704+0000","last_undegraded":"2024-09-16T04:43:00.150704+0000","last_fullsized":"2024-09-16T04:43:00.150704+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:00.461060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":364,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T04:42:57.616300+0000","last_change":"2024-09-16T04:42:57.617205+0000","last_active":"2024-09-16T04:42:57.616300+0000","last_peered":"2024-09-16T04:42:57.616300+0000","last_clean":"2024-09-16T04:42:57.616300+0000","last_became_active":"2024-09-16T04:42:53.798280+0000","last_became_peered":"2024-09-16T04:42:53.798280+0000","last_unstale":"2024-09-16T04:42:57.616300+0000","last_undegraded":"2024-09-16T04:42:57.616300+0000","last_fullsized":"2024-09-16T04:42:57.616300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:18.626574+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.0013274440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,14,12,7,9],"acting":[0,13,1,14,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":181,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.654119+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:42:59.654119+0000","last_peered":"2024-09-16T04:42:59.654119+0000","last_clean":"2024-09-16T04:42:59.654119+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:42:59.654119+0000","last_undegraded":"2024-09-16T04:42:59.654119+0000","last_fullsized":"2024-09-16T04:42:59.654119+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":365,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.654083+0000","last_change":"2024-09-16T04:42:57.627200+0000","last_active":"2024-09-16T04:42:59.654083+0000","last_peered":"2024-09-16T04:42:59.654083+0000","last_clean":"2024-09-16T04:42:59.654083+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:42:59.654083+0000","last_undegraded":"2024-09-16T04:42:59.654083+0000","last_fullsized":"2024-09-16T04:42:59.654083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:42:00.916431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.916431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:03.388292+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.0023689520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.646569+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:42:59.646569+0000","last_peered":"2024-09-16T04:42:59.646569+0000","last_clean":"2024-09-16T04:42:59.646569+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:42:59.646569+0000","last_undegraded":"2024-09-16T04:42:59.646569+0000","last_fullsized":"2024-09-16T04:42:59.646569+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"107'31","reported_seq":405,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.654628+0000","last_change":"2024-09-16T04:42:58.139380+0000","last_active":"2024-09-16T04:42:59.654628+0000","last_peered":"2024-09-16T04:42:59.654628+0000","last_clean":"2024-09-16T04:42:59.654628+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:42:59.654628+0000","last_undegraded":"2024-09-16T04:42:59.654628+0000","last_fullsized":"2024-09-16T04:42:59.654628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:59.997918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.997918+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:16.258201+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.30758759000000002,"stat_sum":{"num_bytes":5210112,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":11,"num_write_kb":4496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":187,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.754455+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:42:59.754455+0000","last_peered":"2024-09-16T04:42:59.754455+0000","last_clean":"2024-09-16T04:42:59.754455+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:42:59.754455+0000","last_undegraded":"2024-09-16T04:42:59.754455+0000","last_fullsized":"2024-09-16T04:42:59.754455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"97'84","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.655347+0000","last_change":"2024-09-16T04:42:57.928525+0000","last_active":"2024-09-16T04:42:59.655347+0000","last_peered":"2024-09-16T04:42:59.655347+0000","last_clean":"2024-09-16T04:42:59.655347+0000","last_became_active":"2024-09-16T04:42:53.912853+0000","last_became_peered":"2024-09-16T04:42:53.912853+0000","last_unstale":"2024-09-16T04:42:59.655347+0000","last_undegraded":"2024-09-16T04:42:59.655347+0000","last_fullsized":"2024-09-16T04:42:59.655347+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:14.758130+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.102446947,"stat_sum":{"num_bytes":4980736,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":676,"num_write":36,"num_write_kb":4208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,13,7,0,6],"acting":[10,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.3","version":"97'43","reported_seq":425,"reported_epoch":109,"state":"peering","last_fresh":"2024-09-16T04:42:59.786594+0000","last_change":"2024-09-16T04:43:00.152735+0000","last_active":"2024-09-16T04:42:51.589751+0000","last_peered":"2024-09-16T04:42:59.786594+0000","last_clean":"2024-09-16T04:42:59.786594+0000","last_became_active":"2024-09-16T04:42:18.590615+0000","last_became_peered":"2024-09-16T04:42:55.109378+0000","last_unstale":"2024-09-16T04:42:59.786594+0000","last_undegraded":"2024-09-16T04:42:59.786594+0000","last_fullsized":"2024-09-16T04:42:59.786594+0000","mapping_epoch":102,"log_start":"97'43","ondisk_log_start":"97'43","created":32,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'21","last_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_deep_scrub":"72'21","last_deep_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.317975+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":48,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0.81083347400000005,"stat_sum":{"num_bytes":5120000,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":4592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5668864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,9,2,5,3],"acting":[13,7,4,9,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,4,5,7,9],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1c","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":191,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.656188+0000","last_change":"2024-09-16T04:42:46.312976+0000","last_active":"2024-09-16T04:42:59.656188+0000","last_peered":"2024-09-16T04:42:59.656188+0000","last_clean":"2024-09-16T04:42:59.656188+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:42:59.656188+0000","last_undegraded":"2024-09-16T04:42:59.656188+0000","last_fullsized":"2024-09-16T04:42:59.656188+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:42:46.312734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:46.312734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:49.120009+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":390,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.650245+0000","last_change":"2024-09-16T04:42:57.640829+0000","last_active":"2024-09-16T04:42:59.650245+0000","last_peered":"2024-09-16T04:42:59.650245+0000","last_clean":"2024-09-16T04:42:59.650245+0000","last_became_active":"2024-09-16T04:42:53.908802+0000","last_became_peered":"2024-09-16T04:42:53.908802+0000","last_unstale":"2024-09-16T04:42:59.650245+0000","last_undegraded":"2024-09-16T04:42:59.650245+0000","last_fullsized":"2024-09-16T04:42:59.650245+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.089771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.089771+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:34.499470+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.001566668,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11,1,5,6,4],"acting":[14,2,11,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":182,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T04:43:00.148828+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:43:00.148828+0000","last_peered":"2024-09-16T04:43:00.148828+0000","last_clean":"2024-09-16T04:43:00.148828+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:43:00.148828+0000","last_undegraded":"2024-09-16T04:43:00.148828+0000","last_fullsized":"2024-09-16T04:43:00.148828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"105'59","reported_seq":450,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.757062+0000","last_change":"2024-09-16T04:42:57.725943+0000","last_active":"2024-09-16T04:42:59.757062+0000","last_peered":"2024-09-16T04:42:59.757062+0000","last_clean":"2024-09-16T04:42:59.757062+0000","last_became_active":"2024-09-16T04:42:18.991363+0000","last_became_peered":"2024-09-16T04:42:18.991363+0000","last_unstale":"2024-09-16T04:42:59.757062+0000","last_undegraded":"2024-09-16T04:42:59.757062+0000","last_fullsized":"2024-09-16T04:42:59.757062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'35","last_scrub_stamp":"2024-09-16T04:42:24.144068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.144068+0000","objects_scrubbed":2,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:42.650523+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0.0014621390000000001,"stat_sum":{"num_bytes":5275648,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.655425+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:42:59.655425+0000","last_peered":"2024-09-16T04:42:59.655425+0000","last_clean":"2024-09-16T04:42:59.655425+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:42:59.655425+0000","last_undegraded":"2024-09-16T04:42:59.655425+0000","last_fullsized":"2024-09-16T04:42:59.655425+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"59'15","reported_seq":403,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T04:43:00.148657+0000","last_change":"2024-09-16T04:42:57.756759+0000","last_active":"2024-09-16T04:43:00.148657+0000","last_peered":"2024-09-16T04:43:00.148657+0000","last_clean":"2024-09-16T04:43:00.148657+0000","last_became_active":"2024-09-16T04:42:54.101490+0000","last_became_peered":"2024-09-16T04:42:54.101490+0000","last_unstale":"2024-09-16T04:43:00.148657+0000","last_undegraded":"2024-09-16T04:43:00.148657+0000","last_fullsized":"2024-09-16T04:43:00.148657+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'15","last_scrub_stamp":"2024-09-16T04:42:26.595325+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.595325+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:32.528406+0000","scrub_duration":967,"objects_trimmed":0,"snaptrim_duration":0.104938434,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,12,1,11],"acting":[13,2,3,6,12,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.3","version":"26'1","reported_seq":190,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.786240+0000","last_change":"2024-09-16T04:42:54.301521+0000","last_active":"2024-09-16T04:42:59.786240+0000","last_peered":"2024-09-16T04:42:59.786240+0000","last_clean":"2024-09-16T04:42:59.786240+0000","last_became_active":"2024-09-16T04:42:54.097360+0000","last_became_peered":"2024-09-16T04:42:54.097360+0000","last_unstale":"2024-09-16T04:42:59.786240+0000","last_undegraded":"2024-09-16T04:42:59.786240+0000","last_fullsized":"2024-09-16T04:42:59.786240+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,13],"acting":[5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":385,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T04:43:00.248905+0000","last_change":"2024-09-16T04:42:36.062121+0000","last_active":"2024-09-16T04:43:00.248905+0000","last_peered":"2024-09-16T04:43:00.248905+0000","last_clean":"2024-09-16T04:43:00.248905+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:43:00.248905+0000","last_undegraded":"2024-09-16T04:43:00.248905+0000","last_fullsized":"2024-09-16T04:43:00.248905+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:42:36.061866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:42:36.061866+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:37.369149+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"107'35","reported_seq":451,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.986121+0000","last_change":"2024-09-16T04:42:58.357253+0000","last_active":"2024-09-16T04:42:59.986121+0000","last_peered":"2024-09-16T04:42:59.986121+0000","last_clean":"2024-09-16T04:42:59.986121+0000","last_became_active":"2024-09-16T04:42:57.944513+0000","last_became_peered":"2024-09-16T04:42:57.944513+0000","last_unstale":"2024-09-16T04:42:59.986121+0000","last_undegraded":"2024-09-16T04:42:59.986121+0000","last_fullsized":"2024-09-16T04:42:59.986121+0000","mapping_epoch":106,"log_start":"44'8","ondisk_log_start":"44'8","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-09-16T04:42:13.576020+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:13.576020+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":12,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:39.838401+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0.310307373,"stat_sum":{"num_bytes":1982464,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1982464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,10,6],"acting":[5,0,2,1,14,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.9","version":"50'30","reported_seq":417,"reported_epoch":108,"state":"peering","last_fresh":"2024-09-16T04:42:59.379255+0000","last_change":"2024-09-16T04:42:59.651167+0000","last_active":"2024-09-16T04:42:59.651167+0000","last_peered":"2024-09-16T04:42:59.379255+0000","last_clean":"2024-09-16T04:42:50.803821+0000","last_became_active":"2024-09-16T04:42:55.006952+0000","last_became_peered":"2024-09-16T04:42:55.006952+0000","last_unstale":"2024-09-16T04:42:59.379255+0000","last_undegraded":"2024-09-16T04:42:59.379255+0000","last_fullsized":"2024-09-16T04:42:54.596572+0000","mapping_epoch":109,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'30","last_scrub_stamp":"2024-09-16T04:42:10.554025+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.554025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.001395947,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,4,8,10],"acting":[14,1,0,3,4,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3,4,8,10],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1c","length":"1"}]},{"pgid":"3.a","version":"97'77","reported_seq":464,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.656029+0000","last_change":"2024-09-16T04:42:57.825659+0000","last_active":"2024-09-16T04:42:59.656029+0000","last_peered":"2024-09-16T04:42:59.656029+0000","last_clean":"2024-09-16T04:42:59.656029+0000","last_became_active":"2024-09-16T04:42:18.694273+0000","last_became_peered":"2024-09-16T04:42:18.694273+0000","last_unstale":"2024-09-16T04:42:59.656029+0000","last_undegraded":"2024-09-16T04:42:59.656029+0000","last_fullsized":"2024-09-16T04:42:59.656029+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'41","last_scrub_stamp":"2024-09-16T04:42:25.530017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.530017+0000","objects_scrubbed":5,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:33.302427+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.100560026,"stat_sum":{"num_bytes":360448,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":1372,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.b","version":"105'83","reported_seq":460,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.654317+0000","last_change":"2024-09-16T04:42:57.630747+0000","last_active":"2024-09-16T04:42:59.654317+0000","last_peered":"2024-09-16T04:42:59.654317+0000","last_clean":"2024-09-16T04:42:59.654317+0000","last_became_active":"2024-09-16T04:42:18.688040+0000","last_became_peered":"2024-09-16T04:42:18.688040+0000","last_unstale":"2024-09-16T04:42:59.654317+0000","last_undegraded":"2024-09-16T04:42:59.654317+0000","last_fullsized":"2024-09-16T04:42:59.654317+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'58","last_scrub_stamp":"2024-09-16T04:42:25.280206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.280206+0000","objects_scrubbed":6,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:40.681604+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0.0035392930000000002,"stat_sum":{"num_bytes":4964352,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":9112,"num_write":6,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.c","version":"107'119","reported_seq":482,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T04:42:58.138226+0000","last_change":"2024-09-16T04:42:58.240919+0000","last_active":"2024-09-16T04:42:58.138226+0000","last_peered":"2024-09-16T04:42:58.138226+0000","last_clean":"2024-09-16T04:42:58.138226+0000","last_became_active":"2024-09-16T04:42:18.798872+0000","last_became_peered":"2024-09-16T04:42:18.798872+0000","last_unstale":"2024-09-16T04:42:58.138226+0000","last_undegraded":"2024-09-16T04:42:58.138226+0000","last_fullsized":"2024-09-16T04:42:58.138226+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:35.566994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.566994+0000","objects_scrubbed":6,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:51.207793+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0.414655147,"stat_sum":{"num_bytes":10485760,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":17,"num_write_kb":6864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.d","version":"105'203","reported_seq":563,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.952120+0000","last_change":"2024-09-16T04:42:57.825072+0000","last_active":"2024-09-16T04:42:59.952120+0000","last_peered":"2024-09-16T04:42:59.952120+0000","last_clean":"2024-09-16T04:42:59.952120+0000","last_became_active":"2024-09-16T04:42:19.196254+0000","last_became_peered":"2024-09-16T04:42:19.196254+0000","last_unstale":"2024-09-16T04:42:59.952120+0000","last_undegraded":"2024-09-16T04:42:59.952120+0000","last_fullsized":"2024-09-16T04:42:59.952120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'134","last_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_deep_scrub":"74'134","last_deep_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.664276+0000","objects_scrubbed":9,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:38.663642+0000","scrub_duration":1071,"objects_trimmed":0,"snaptrim_duration":0.20199710000000001,"stat_sum":{"num_bytes":6995968,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":7,"num_read_kb":4921,"num_write":28,"num_write_kb":1936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.e","version":"92'88","reported_seq":462,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.784109+0000","last_change":"2024-09-16T04:42:57.725578+0000","last_active":"2024-09-16T04:42:59.784109+0000","last_peered":"2024-09-16T04:42:59.784109+0000","last_clean":"2024-09-16T04:42:59.784109+0000","last_became_active":"2024-09-16T04:42:18.700111+0000","last_became_peered":"2024-09-16T04:42:18.700111+0000","last_unstale":"2024-09-16T04:42:59.784109+0000","last_undegraded":"2024-09-16T04:42:59.784109+0000","last_fullsized":"2024-09-16T04:42:59.784109+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:34.646929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.646929+0000","objects_scrubbed":6,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:51.365959+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.0021028940000000001,"stat_sum":{"num_bytes":5193728,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":9985,"num_write":1,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.f","version":"97'94","reported_seq":463,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T04:42:57.827638+0000","last_change":"2024-09-16T04:42:57.928111+0000","last_active":"2024-09-16T04:42:57.827638+0000","last_peered":"2024-09-16T04:42:57.827638+0000","last_clean":"2024-09-16T04:42:57.827638+0000","last_became_active":"2024-09-16T04:42:18.793319+0000","last_became_peered":"2024-09-16T04:42:18.793319+0000","last_unstale":"2024-09-16T04:42:57.827638+0000","last_undegraded":"2024-09-16T04:42:57.827638+0000","last_fullsized":"2024-09-16T04:42:57.827638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'74","last_scrub_stamp":"2024-09-16T04:42:34.345034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.345034+0000","objects_scrubbed":7,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.660191+0000","scrub_duration":719,"objects_trimmed":0,"snaptrim_duration":0.20213098900000001,"stat_sum":{"num_bytes":3899392,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":12161,"num_write":5,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.10","version":"108'43","reported_seq":99,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T04:43:00.148849+0000","last_change":"2024-09-16T04:42:59.465896+0000","last_active":"2024-09-16T04:43:00.148849+0000","last_peered":"2024-09-16T04:43:00.148849+0000","last_clean":"2024-09-16T04:43:00.148849+0000","last_became_active":"2024-09-16T04:42:53.917288+0000","last_became_peered":"2024-09-16T04:42:53.917288+0000","last_unstale":"2024-09-16T04:43:00.148849+0000","last_undegraded":"2024-09-16T04:43:00.148849+0000","last_fullsized":"2024-09-16T04:43:00.148849+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:42:24.882865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.882865+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:42.847568+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0.396969236,"stat_sum":{"num_bytes":3162112,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":6,"num_read_kb":1682,"num_write":7,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3735552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,0,11],"acting":[13,6,7,4,8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.11","version":"107'60","reported_seq":381,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T04:42:59.655421+0000","last_change":"2024-09-16T04:42:58.030327+0000","last_active":"2024-09-16T04:42:59.655421+0000","last_peered":"2024-09-16T04:42:59.655421+0000","last_clean":"2024-09-16T04:42:59.655421+0000","last_became_active":"2024-09-16T04:42:53.807035+0000","last_became_peered":"2024-09-16T04:42:53.807035+0000","last_unstale":"2024-09-16T04:42:59.655421+0000","last_undegraded":"2024-09-16T04:42:59.655421+0000","last_fullsized":"2024-09-16T04:42:59.655421+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'28","last_scrub_stamp":"2024-09-16T04:42:10.158979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.158979+0000","objects_scrubbed":3,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:36.334840+0000","scrub_duration":855,"objects_trimmed":1,"snaptrim_duration":0.30489170799999998,"stat_sum":{"num_bytes":2179072,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5233,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2572288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,13,9,5,6,1],"acting":[10,11,13,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.12","version":"105'38","reported_seq":343,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T04:43:01.158722+0000","last_change":"2024-09-16T04:42:57.718276+0000","last_active":"2024-09-16T04:43:01.158722+0000","last_peered":"2024-09-16T04:43:01.158722+0000","last_clean":"2024-09-16T04:43:01.158722+0000","last_became_active":"2024-09-16T04:42:53.712428+0000","last_became_peered":"2024-09-16T04:42:53.712428+0000","last_unstale":"2024-09-16T04:43:01.158722+0000","last_undegraded":"2024-09-16T04:43:01.158722+0000","last_fullsized":"2024-09-16T04:43:01.158722+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'12","last_scrub_stamp":"2024-09-16T04:42:10.115913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.115913+0000","objects_scrubbed":1,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:20.386327+0000","scrub_duration":611,"objects_trimmed":0,"snaptrim_duration":0.10218153200000001,"stat_sum":{"num_bytes":2555904,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3745,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2555904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,12],"acting":[9,7,8,6,4,3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.13","version":"36'3","reported_seq":272,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T04:42:57.828495+0000","last_change":"2024-09-16T04:42:57.829230+0000","last_active":"2024-09-16T04:42:57.828495+0000","last_peered":"2024-09-16T04:42:57.828495+0000","last_clean":"2024-09-16T04:42:57.828495+0000","last_became_active":"2024-09-16T04:42:04.415210+0000","last_became_peered":"2024-09-16T04:42:04.415210+0000","last_unstale":"2024-09-16T04:42:57.828495+0000","last_undegraded":"2024-09-16T04:42:57.828495+0000","last_fullsized":"2024-09-16T04:42:57.828495+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:42:10.690368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.690368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:33.500771+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.10308851600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,0,4,10],"acting":[11,7,9,13,0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":35,"stat_sum":{"num_bytes":118718464,"num_objects":95,"num_object_clones":48,"num_object_copies":665,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":1,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":7,"num_read":186,"num_read_kb":96256,"num_write":326,"num_write_kb":68600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":77922304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250220544,"data_stored":316772352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1797,"ondisk_log_size":1797,"up":245,"acting":245,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247832,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50780,"kb_used_data":15428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321060,"statfs":{"total":96636764160,"available":96584765440,"internally_reserved":0,"allocated":15798272,"data_stored":19355038,"data_compressed":295368,"data_compressed_allocated":401408,"data_compressed_original":1232896,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247832,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52128,"kb_used_data":16776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319712,"statfs":{"total":96636764160,"available":96583385088,"internally_reserved":0,"allocated":17178624,"data_stored":19358590,"data_compressed":263423,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247832,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54512,"kb_used_data":19160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94317328,"statfs":{"total":96636764160,"available":96580943872,"internally_reserved":0,"allocated":19619840,"data_stored":22801054,"data_compressed":260352,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313238,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53844,"kb_used_data":18492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94317996,"statfs":{"total":96636764160,"available":96581627904,"internally_reserved":0,"allocated":18935808,"data_stored":25072076,"data_compressed":251150,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313238,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50664,"kb_used_data":19476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321176,"statfs":{"total":96636764160,"available":96584884224,"internally_reserved":0,"allocated":19943424,"data_stored":26063308,"data_compressed":251150,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280535,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44444,"kb_used_data":13256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327396,"statfs":{"total":96636764160,"available":96591253504,"internally_reserved":0,"allocated":13574144,"data_stored":18091198,"data_compressed":292297,"data_compressed_allocated":397312,"data_compressed_original":1224704,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280535,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46808,"kb_used_data":15620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325032,"statfs":{"total":96636764160,"available":96588832768,"internally_reserved":0,"allocated":15994880,"data_stored":18954929,"data_compressed":260352,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280535,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54968,"kb_used_data":19620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316872,"statfs":{"total":96636764160,"available":96580476928,"internally_reserved":0,"allocated":20090880,"data_stored":24967838,"data_compressed":260352,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280535,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41840,"kb_used_data":10648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330000,"statfs":{"total":96636764160,"available":96593920000,"internally_reserved":0,"allocated":10903552,"data_stored":15438588,"data_compressed":257287,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280534,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46376,"kb_used_data":15180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325464,"statfs":{"total":96636764160,"available":96589275136,"internally_reserved":0,"allocated":15544320,"data_stored":18154236,"data_compressed":257287,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280535,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52856,"kb_used_data":17508,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318984,"statfs":{"total":96636764160,"available":96582639616,"internally_reserved":0,"allocated":17928192,"data_stored":21138078,"data_compressed":260352,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280534,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57644,"kb_used_data":26448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314196,"statfs":{"total":96636764160,"available":96577736704,"internally_reserved":0,"allocated":27082752,"data_stored":32660126,"data_compressed":260352,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280535,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47100,"kb_used_data":15904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324740,"statfs":{"total":96636764160,"available":96588533760,"internally_reserved":0,"allocated":16285696,"data_stored":21242620,"data_compressed":257287,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41440,"kb_used_data":10244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330400,"statfs":{"total":96636764160,"available":96594329600,"internally_reserved":0,"allocated":10489856,"data_stored":12108559,"data_compressed":257287,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280535,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52848,"kb_used_data":17504,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318992,"statfs":{"total":96636764160,"available":96582647808,"internally_reserved":0,"allocated":17924096,"data_stored":22881020,"data_compressed":257287,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17702912,"data_stored":24141824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18632704,"data_stored":22122496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12242944,"data_stored":16601088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14721024,"data_stored":17993728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18821120,"data_stored":24006656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10166272,"data_stored":15007744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14811136,"data_stored":17727488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16658432,"data_stored":20176896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":26071040,"data_stored":31956992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":15896576,"data_stored":18386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":15028224,"data_stored":20291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":19496960,"data_stored":25473024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":9490432,"data_stored":11419648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":14716928,"data_stored":18116608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14368768,"data_stored":19214336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":11395072,"data_stored":14135296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:02.974 INFO:tasks.ceph.ceph_manager.ceph:pg 3.1d in trimming, state: active+clean+snaptrim 2024-09-16T04:43:02.974 INFO:tasks.ceph.ceph_manager.ceph:unique_pool_0 still trimming, waiting 2024-09-16T04:43:03.040 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:03.034+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:03.072 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:03.072 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:43:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:03.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:43:03.141 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:03.135+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:03.203 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:03.203 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:03.204 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:43:03.204 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:43:03.204 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:03.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:43:03.242 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:43:03.238+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:43:03.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:108 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 77.976411857000002, 2024-09-16T04:43:03.342 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60644887800000002, 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 108 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.343 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.344 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362769+0000", 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.364e-06 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362776+0000", 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9789999999999996e-06 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362795+0000", 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8569000000000001e-05 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362811+0000", 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6084999999999999e-05 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369499+0000", 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066885180000000001 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369745+0000", 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024616900000000002 2024-09-16T04:43:03.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669288+0000", 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29954281100000002 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669425+0000", 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013708200000000001 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.347 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.969217+0000", 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29979130100000001 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:507 3.es0 3:7777e846:::smithi16530038-17:head [copy-from ver 21] snapc 10=[10,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 49.067356918999998, 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.526543015, 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.348 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 507 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.349 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271824+0000", 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7990000000000001e-06 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271832+0000", 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271852+0000", 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9194999999999998e-05 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.371991+0000", 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10013957800000001 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372175+0000", 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000183325 2024-09-16T04:43:03.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372198+0000", 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3306e-05 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472740+0000", 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100542016 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.352 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472902+0000", 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016220399999999999 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692422+0000", 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21951984699999999 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.796322+0000", 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10389983699999999 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798260+0000", 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001938103 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798366+0000", 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010582900000000001 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.354 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:582 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 43.999810863, 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69089629399999997, 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 582 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.355 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339370+0000", 2024-09-16T04:43:03.356 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339374+0000", 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0799999999999999e-06 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339387+0000", 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3464e-05 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339414+0000", 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7223e-05 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446804+0000", 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.107389545 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446957+0000", 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000153389 2024-09-16T04:43:03.358 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727239+0000", 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28028220199999998 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727462+0000", 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022226099999999999 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.030265+0000", 2024-09-16T04:43:03.359 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280332199999999 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:613 3.es0 3:7777e846:::smithi16530038-17:head [append 0~589824] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.978960375, 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55445315100000003, 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.360 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 613 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.361 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360223+0000", 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3730000000000001e-06 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361140+0000", 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00091743300000000005 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.362 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361178+0000", 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7691999999999998e-05 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361197+0000", 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.891e-05 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361351+0000", 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015419300000000001 2024-09-16T04:43:03.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361371+0000", 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9726000000000001e-05 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366548+0000", 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051778730000000004 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.364 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366714+0000", 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000165329 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.369378+0000", 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026643159999999999 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.372406+0000", 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003027891 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.365 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914520+0000", 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54211425499999999 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914672+0000", 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015216000000000001 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.366 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:614 3.es0 3:7777e846:::smithi16530038-17:head [append 0~540672] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.974987329000001, 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55075342599999999, 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 614 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.367 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364194+0000", 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.26e-06 2024-09-16T04:43:03.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365041+0000", 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00084621499999999999 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365073+0000", 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2719000000000003e-05 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.369 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465237+0000", 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100163671 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465392+0000", 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015528300000000001 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465730+0000", 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000338005 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.370 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.467540+0000", 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001809583 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.469612+0000", 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020719760000000001 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914847+0000", 2024-09-16T04:43:03.371 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.44523504899999999 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914946+0000", 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8665000000000005e-05 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:617 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:43:03.372 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.7724732, 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71993709900000002, 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 617 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:43:03.373 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566707+0000", 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.374 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566713+0000", 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.784e-06 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566725+0000", 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.173e-05 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566732+0000", 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6860000000000006e-06 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.375 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579439+0000", 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012706935000000001 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579759+0000", 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032009600000000002 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.376 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074376+0000", 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49461618299999999 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074520+0000", 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014481800000000001 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.286643+0000", 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212123017 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.377 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:696 3.10s0 3:0fc9379d:::smithi16530038-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 33.914470966000003, 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69507003199999995, 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.378 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 696 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.379 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424713+0000", 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5599999999999998e-06 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.380 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424724+0000", 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0906e-05 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424732+0000", 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6499999999999996e-06 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502418+0000", 2024-09-16T04:43:03.381 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077686693000000001 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502530+0000", 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000111644 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917065+0000", 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41453470199999998 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.382 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917391+0000", 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032674700000000001 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.119779+0000", 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202387337 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:805 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:43:03.383 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.834116227999999, 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51150757400000002, 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 805 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.384 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505067+0000", 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8139999999999998e-06 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.385 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505082+0000", 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4453e-05 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505124+0000", 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2849999999999998e-05 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505148+0000", 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3062999999999999e-05 2024-09-16T04:43:03.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611642+0000", 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106494218 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611801+0000", 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000159338 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.613368+0000", 2024-09-16T04:43:03.387 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001566462 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.711952+0000", 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098584630000000006 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016439+0000", 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30448657000000001 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016571+0000", 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013217599999999999 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:806 3.10s0 3:0fc9379d:::smithi16530038-35:head [rollback 11] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.833982898999999, 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89166837600000004, 2024-09-16T04:43:03.389 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 806 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.390 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505198+0000", 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7239999999999999e-06 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505210+0000", 2024-09-16T04:43:03.391 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1688999999999999e-05 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505232+0000", 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1903999999999999e-05 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505249+0000", 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7036000000000001e-05 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.392 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017365+0000", 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.512115616 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017525+0000", 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016080200000000001 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.018648+0000", 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011226109999999999 2024-09-16T04:43:03.393 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021794+0000", 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031461660000000002 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396721+0000", 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.37492636899999998 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.394 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396865+0000", 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014445899999999999 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:807 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 22.833901575999999, 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89196936699999996, 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.395 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 807 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.396 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505279+0000", 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505285+0000", 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0399999999999998e-06 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.397 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505308+0000", 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2595999999999998e-05 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505323+0000", 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5342999999999999e-05 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019548+0000", 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51422475999999995 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.398 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019822+0000", 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027388100000000002 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021062+0000", 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001239614 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.122291+0000", 2024-09-16T04:43:03.399 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101229427 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397057+0000", 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27476627300000001 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397247+0000", 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000190096 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.400 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:809 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~540672] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.821887165, 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70063091600000005, 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 809 2024-09-16T04:43:03.401 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.402 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517297+0000", 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8899999999999998e-06 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518377+0000", 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010801750000000001 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518421+0000", 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3114000000000001e-05 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.403 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518443+0000", 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2782e-05 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518619+0000", 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017519200000000001 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518647+0000", 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8053999999999999e-05 2024-09-16T04:43:03.404 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698939+0000", 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180292338 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.699242+0000", 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000302911 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.702023+0000", 2024-09-16T04:43:03.405 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027812560000000002 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813606+0000", 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11158314900000001 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.406 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217721+0000", 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40411436000000001 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217923+0000", 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000202695 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:810 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~720896] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:43:03.407 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.815882187, 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81676015000000002, 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 810 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.408 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523300+0000", 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1140000000000002e-06 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.409 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524411+0000", 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001110307 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524443+0000", 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2067000000000003e-05 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524458+0000", 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5223e-05 2024-09-16T04:43:03.410 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524644+0000", 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018582900000000001 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524656+0000", 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1705999999999999e-05 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.704974+0000", 2024-09-16T04:43:03.411 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180318379 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.705144+0000", 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017024499999999999 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.706863+0000", 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00171842 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.412 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813610+0000", 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10674731 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.339927+0000", 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52631677099999996 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340058+0000", 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130779 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.413 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.414 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.414 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.416 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:811 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~786432] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:03.416 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.809229711, 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81051717700000003, 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 811 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.417 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529953+0000", 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.824e-06 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.418 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531687+0000", 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017341699999999999 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531721+0000", 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3756e-05 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.419 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531744+0000", 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531894+0000", 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000150505 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:03.420 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531914+0000", 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0137e-05 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808146+0000", 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.276231536 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808322+0000", 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000175899 2024-09-16T04:43:03.421 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.810020+0000", 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016987230000000001 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813806+0000", 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0037855419999999998 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340306+0000", 2024-09-16T04:43:03.422 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52649966299999995 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340467+0000", 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016146800000000001 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:812 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~49152] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:43:03.423 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.807397114, 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80902035800000005, 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 812 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:43:03.424 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531784+0000", 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.796e-06 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.425 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531896+0000", 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011207 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531918+0000", 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1412000000000001e-05 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531934+0000", 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6663999999999998e-05 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.426 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698290+0000", 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16635518999999999 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698310+0000", 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0587000000000001e-05 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811428+0000", 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11311787800000001 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.427 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811601+0000", 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017340500000000001 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.812624+0000", 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001022659 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815470+0000", 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002845752 2024-09-16T04:43:03.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340654+0000", 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52518383899999999 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340803+0000", 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014910600000000001 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:813 3.22s0 3:456a3a59:::smithi16530038-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.807218623000001, 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80925618399999999, 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 813 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.430 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531962+0000", 2024-09-16T04:43:03.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4339999999999999e-06 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531969+0000", 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7329999999999997e-06 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531987+0000", 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7467000000000001e-05 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632132+0000", 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10014582499999999 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813864+0000", 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18173177200000001 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814024+0000", 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016021500000000001 2024-09-16T04:43:03.433 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814762+0000", 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00073760799999999995 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.816600+0000", 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001837679 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341069+0000", 2024-09-16T04:43:03.434 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52446895500000001 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341217+0000", 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000148496 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:814 3.22s0 3:456a3a59:::smithi16530038-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:43:03.435 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 21.707013781000001, 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.878598294, 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 814 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.436 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632168+0000", 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:43:03.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632177+0000", 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9500000000000007e-06 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632196+0000", 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9296e-05 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632207+0000", 2024-09-16T04:43:03.438 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.137e-05 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815302+0000", 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18309505200000001 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815453+0000", 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000151065 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.439 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342052+0000", 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52659867599999999 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342192+0000", 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013975200000000001 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.510764+0000", 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.168572213 2024-09-16T04:43:03.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1es0 e100,97 3:78f34faf:::smithi16530038-13:f,90'68)", 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.151723957, 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.63827547699999998, 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.441 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.442 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187458+0000", 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.12e-06 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187470+0000", 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2356e-05 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187502+0000", 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2135999999999999e-05 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.443 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187519+0000", 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6730000000000001e-05 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.823533+0000", 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.63601388800000003 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.825731+0000", 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002198247 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.444 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1es0 e100,97 3:78f34faf:::smithi16530038-13:11,92'72)", 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 12.041120832000001, 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.537458137, 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.445 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298060+0000", 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.607e-06 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.446 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298071+0000", 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.03e-05 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298103+0000", 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2308999999999998e-05 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298119+0000", 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5642999999999999e-05 2024-09-16T04:43:03.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.834381+0000", 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.53626253700000004 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.835517+0000", 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001135741 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.448 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:958 3.1es0 3:78f34faf:::smithi16530038-13:head [rollback 1d] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.7438540089999996, 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54156909600000003, 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 958 2024-09-16T04:43:03.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.450 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595327+0000", 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.257e-06 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595333+0000", 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0959999999999997e-06 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595353+0000", 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9656999999999998e-05 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.451 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595375+0000", 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1982999999999999e-05 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.722826+0000", 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12745103099999999 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.723141+0000", 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031585100000000002 2024-09-16T04:43:03.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724986+0000", 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00184425 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.831289+0000", 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106303545 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.136810+0000", 2024-09-16T04:43:03.453 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30552102599999997 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.136895+0000", 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4400000000000005e-05 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:959 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 8.7437831090000007, 2024-09-16T04:43:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54178843700000001, 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 959 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595398+0000", 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.223e-06 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595404+0000", 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8239999999999996e-06 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595424+0000", 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0876000000000001e-05 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595433+0000", 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9099999999999994e-06 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.827455+0000", 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23202157200000001 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.827657+0000", 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000201709 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828666+0000", 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010089389999999999 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.831586+0000", 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029207149999999999 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.137124+0000", 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30553806500000003 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.137185+0000", 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0603999999999998e-05 2024-09-16T04:43:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:03.460 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:03.461 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:43:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:43:03.462 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:43:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:43:03.463 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:43:03.463 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:03.458+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:03.477 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:43:03.477 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:43:03.498 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:43:03.499 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:43:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:43:03.516 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:43:03.536 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:43:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:43:03.556 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:43:03.556 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:43:03.562 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:03.558+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:03.576 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:43:03.576 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:43:03.589 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:03.589 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:03.597 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:43:03.597 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:43:03.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:43:03.621 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:43:03.621 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:43:03.624 INFO:teuthology.orchestra.run.smithi165.stdout:reshard success 2024-09-16T04:43:03.639 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:43:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:43:03.662 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:03.658+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:03.663 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:43:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:43:03.683 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:43:03.684 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:43:03.706 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:43:03.707 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:43:03.727 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:03.727 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:03.729 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:43:03.730 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:43:03.730 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.15 2024-09-16T04:43:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:43:03.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:43:03.742 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:43:03.786 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:43:03.783+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:43:03.861 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:03.861 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:03.863 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:03.858+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:03.876 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:43:03.964 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:03.959+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:04.043 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:04.043 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:04.053 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.15 2024-09-16T04:43:04.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-15 fsck 2024-09-16T04:43:04.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:43:04.097 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:43:04.095+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:43:04.166 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:04.163+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:04.175 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:04.176 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:04.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:43:04.266 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:43:04.262+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:43:04.360 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:04.360 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:04.368 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:04.366+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:04.374 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:43:04.469 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:43:04.462+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:43:04.496 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:04.496 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:04.510 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:43:04.570 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:43:04.567+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:43:04.628 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:04.629 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:04.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:43:04.671 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:04.666+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:04.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:43:04.767 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:04.768 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:04.772 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:04.770+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:04.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:43:04.872 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:04.867+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:04.913 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:04.913 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:04.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:43:04.974 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:04.970+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:05.075 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:05.071+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:05.103 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:05.103 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:05.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:43:05.176 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:05.170+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:05.247 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:05.248 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:05.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:43:05.277 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:43:05.274+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:43:05.377 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:05.374+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:05.390 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:05.390 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:05.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:43:05.479 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:05.475+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:05.529 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:05.529 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:05.529 INFO:teuthology.orchestra.run.smithi165.stdout:fsck success 2024-09-16T04:43:05.540 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-16T04:43:05.541 INFO:tasks.thrashosds.thrasher:Reviving osd 15 2024-09-16T04:43:05.541 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-16T04:43:05.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-16T04:43:05.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:43:05.580 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:43:05.574+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:43:05.586 INFO:tasks.ceph.osd.15:Started 2024-09-16T04:43:05.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:43:05.672 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-09-16T04:43:05.682 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:05.674+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:05.698 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:0 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:1 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:2 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:config diff 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:config diff get 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:config get 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:config help [] 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:config set ... 2024-09-16T04:43:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2024-09-16T04:43:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:43:05.702 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-15, ['dump_ops_in_flight'] 2024-09-16T04:43:05.702 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:05.704 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:05.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:43:05.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:43:05.782 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:05.778+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:05.883 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:05.883 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:05.884 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:43:05.878+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:43:05.892 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:43:05.892 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:43:05.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:43:05.913 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:43:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:43:05.935 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:43:05.935 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:43:05.954 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:43:05.954 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:43:05.972 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:43:05.972 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:43:05.984 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:05.979+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:05.994 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:43:05.994 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:43:06.014 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:06.015 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:06.015 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:43:06.015 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:43:06.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:43:06.037 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:43:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:43:06.057 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:43:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:43:06.081 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:43:06.082 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:43:06.102 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:43:06.102 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:43:06.117 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:06.115+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:06.123 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:43:06.124 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:43:06.143 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:43:06.143 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:43:06.167 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:43:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:43:06.186 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:06.183+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:06.191 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:43:06.192 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:43:06.192 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.15 2024-09-16T04:43:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:43:06.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:43:06.200 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:154 3.fs0 3:f6276c08:::smithi16530038-20:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.986286437000004, 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2626387860000001, 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.201 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 154 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210497+0000", 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210502+0000", 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9139999999999999e-06 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210519+0000", 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6969999999999998e-05 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210555+0000", 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6099000000000001e-05 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210573+0000", 2024-09-16T04:43:06.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7770000000000001e-05 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210738+0000", 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016486499999999999 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210962+0000", 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022410600000000001 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.269739+0000", 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.058777538 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.272567+0000", 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028276550000000001 2024-09-16T04:43:06.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.472907+0000", 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20033962699999999 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.473136+0000", 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000229242 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.207 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:155 3.fs0 3:f6276c08:::smithi16530038-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.986170427999994, 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.061292586, 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 155 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210613+0000", 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210614+0000", 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7000000000000003e-07 2024-09-16T04:43:06.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210619+0000", 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210636+0000", 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7436999999999999e-05 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.210645+0000", 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.816e-06 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211299+0000", 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065386899999999996 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.211385+0000", 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5852000000000004e-05 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271332+0000", 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.05994666 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271584+0000", 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025255399999999999 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.271906+0000", 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032151100000000002 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:169 3.cs0 3:34145aa0:::smithi16530038-26:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.300106811000006, 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87942165800000005, 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 169 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896677+0000", 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896679+0000", 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3070000000000001e-06 2024-09-16T04:43:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896692+0000", 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2785999999999999e-05 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896719+0000", 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7540000000000001e-05 2024-09-16T04:43:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.896735+0000", 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5897e-05 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998807+0000", 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102071833 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.998930+0000", 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012282600000000001 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.572766+0000", 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57383636000000005 2024-09-16T04:43:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.574732+0000", 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019661589999999999 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.775926+0000", 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20119392799999999 2024-09-16T04:43:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:47.776098+0000", 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172022 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:207 3.fs0 3:f1175b03:::smithi16530038-29:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 78.096375241999993, 2024-09-16T04:43:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.69161456600000004, 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 207 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100408+0000", 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100411+0000", 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100423+0000", 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.202e-05 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100454+0000", 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0895999999999997e-05 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100469+0000", 2024-09-16T04:43:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5614e-05 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174514+0000", 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.074044911000000005 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.174748+0000", 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023420200000000001 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.687971+0000", 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51322260399999997 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.690498+0000", 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002526978 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791786+0000", 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10128820500000001 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.792023+0000", 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000236772 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:426 3.cs0 3:3dc0a373:::smithi16530038-10:head [append 0~344064] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.860393893999998, 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90840981600000004, 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 426 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336390+0000", 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336391+0000", 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0699999999999996e-07 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336788+0000", 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039739000000000002 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336805+0000", 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6583999999999999e-05 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336814+0000", 2024-09-16T04:43:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4299999999999995e-06 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739098+0000", 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40228381099999999 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.739254+0000", 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156506 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.740433+0000", 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011782450000000001 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943229+0000", 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20279692899999999 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244590+0000", 2024-09-16T04:43:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30136072600000002 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.244799+0000", 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020928799999999999 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:427 3.cs0 3:3dc0a373:::smithi16530038-10:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,3] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:43:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.859952827000001, 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90837145699999999, 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 427 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336831+0000", 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336832+0000", 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7400000000000002e-07 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336837+0000", 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9300000000000002e-06 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336857+0000", 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-05 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336890+0000", 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2852000000000003e-05 2024-09-16T04:43:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841600+0000", 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50471004399999997 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.841814+0000", 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000214589 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.842536+0000", 2024-09-16T04:43:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072138400000000004 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.943303+0000", 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10076771499999999 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245048+0000", 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30174494299999999 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245202+0000", 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015384599999999999 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:428 3.cs0 3:3dc0a373:::smithi16530038-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.859880721000003, 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1109408999999999, 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 428 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336903+0000", 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336904+0000", 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336920+0000", 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6067e-05 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336933+0000", 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:03.336941+0000", 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4600000000000003e-06 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.245817+0000", 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90887578899999999 2024-09-16T04:43:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.246090+0000", 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000273071 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:04.447844+0000", 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20175383 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:519 3.fs0 3:f1175b03:::smithi16530038-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.963440789000003, 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83548977300000005, 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 519 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233343+0000", 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233344+0000", 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6899999999999999e-06 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233352+0000", 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5100000000000001e-06 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233381+0000", 2024-09-16T04:43:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8887e-05 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233398+0000", 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7427000000000001e-05 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233732+0000", 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033334700000000001 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.233749+0000", 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7612999999999999e-05 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236234+0000", 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024850919999999999 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.236334+0000", 2024-09-16T04:43:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100159 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068497+0000", 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83216263000000001 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068656+0000", 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015873499999999999 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.068833+0000", 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017668299999999999 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:534 3.fs0 3:f6276c08:::smithi16530038-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.938048281, 2024-09-16T04:43:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81072121699999999, 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 534 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258735+0000", 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258737+0000", 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.46e-06 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258746+0000", 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3670000000000003e-06 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258767+0000", 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1253000000000001e-05 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.258785+0000", 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8046e-05 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.561788+0000", 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30300253199999999 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.562084+0000", 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029564400000000002 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.662783+0000", 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10069991 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:15.663047+0000", 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026331899999999998 2024-09-16T04:43:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:16.069456+0000", 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40640968599999999 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:892 3.cs0 3:31eec322:::smithi16530038-9:head [append 0~688128] snapc 1e=[1e,1d,1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:46.554483+0000", 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.642300759000001, 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.910856789, 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 892 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.554483+0000", 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.554483+0000", 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.554484+0000", 2024-09-16T04:43:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6950000000000001e-06 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.555102+0000", 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061755700000000002 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.555124+0000", 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2306e-05 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.555136+0000", 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1211e-05 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.255460+0000", 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70032484500000003 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.255770+0000", 2024-09-16T04:43:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000309938 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.257978+0000", 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022072369999999999 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.363681+0000", 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105703227 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.465130+0000", 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101449281 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.465340+0000", 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020949200000000001 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:893 3.cs0 3:31eec322:::smithi16530038-9:head [append 0~245760] snapc 1e=[1e,1d,1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:46.556744+0000", 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.640039083000001, 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.109882466, 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 893 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556744+0000", 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556744+0000", 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556746+0000", 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.57e-06 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556876+0000", 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012994999999999999 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556894+0000", 2024-09-16T04:43:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8473000000000001e-05 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556903+0000", 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3140000000000004e-06 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.360127+0000", 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80322436600000002 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.360440+0000", 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000312551 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.362220+0000", 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001780163 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.564135+0000", 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20191479100000001 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.666362+0000", 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102227027 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.666627+0000", 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026526099999999998 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:894 3.cs0 3:31eec322:::smithi16530038-9:head [setxattr _header (56) in=7b] snapc 1e=[1e,1d,1b,1a,19,18] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:46.556930+0000", 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.639853802000001, 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8184993380000001, 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 894 2024-09-16T04:43:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556930+0000", 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556930+0000", 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556930+0000", 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0100000000000004e-07 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556935+0000", 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0879999999999997e-06 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556942+0000", 2024-09-16T04:43:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4789999999999999e-06 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556947+0000", 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3929999999999999e-06 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.971563+0000", 2024-09-16T04:43:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4146154499999999 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.971745+0000", 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018169200000000001 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.972579+0000", 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083416700000000002 2024-09-16T04:43:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.973952+0000", 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013731699999999999 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.375161+0000", 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40120933600000003 2024-09-16T04:43:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.375429+0000", 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000267862 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:895 3.cs0 3:31eec322:::smithi16530038-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:46.556955+0000", 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.639828179999999, 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1036786709999999, 2024-09-16T04:43:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 895 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556955+0000", 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556955+0000", 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556956+0000", 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.58e-07 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556960+0000", 2024-09-16T04:43:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7340000000000002e-06 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556966+0000", 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2519999999999996e-06 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:46.556970+0000", 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5750000000000002e-06 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.973132+0000", 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.416161435 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.973296+0000", 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016461899999999999 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.376204+0000", 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40290749999999997 2024-09-16T04:43:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.376305+0000", 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100775 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.660634+0000", 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28432932300000002 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:949 3.cs0 3:31eec322:::smithi16530038-9:head [append 0~409600] snapc 23=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:52.656797+0000", 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.539986803, 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70437253399999999, 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 949 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.656797+0000", 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.656797+0000", 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.656799+0000", 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9209999999999999e-06 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657138+0000", 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033954200000000002 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657160+0000", 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1756999999999999e-05 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657170+0000", 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6369999999999994e-06 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.054894+0000", 2024-09-16T04:43:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39772415 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.055043+0000", 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000149479 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.056213+0000", 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001169551 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.058415+0000", 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022022510000000001 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.361076+0000", 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30266109800000002 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.361169+0000", 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3147999999999997e-05 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:950 3.cs0 3:31eec322:::smithi16530038-9:head [setxattr _header (56) in=7b] snapc 23=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:52.657189+0000", 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.539594809, 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70452365100000003, 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 950 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657189+0000", 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657189+0000", 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657189+0000", 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9299999999999997e-07 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657194+0000", 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3710000000000002e-06 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.657200+0000", 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8540000000000002e-06 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.757333+0000", 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100133298 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.057438+0000", 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.300105442 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.057588+0000", 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014921300000000001 2024-09-16T04:43:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.058147+0000", 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055946700000000002 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.058900+0000", 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00075252900000000005 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.361546+0000", 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30264638999999999 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.361712+0000", 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016639399999999999 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:951 3.cs0 3:31eec322:::smithi16530038-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:52.757365+0000", 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.439418635999999, 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80647438800000004, 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 951 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.757365+0000", 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.757365+0000", 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.757367+0000", 2024-09-16T04:43:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.77e-06 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.757375+0000", 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-06 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.757395+0000", 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9582999999999999e-05 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:52.757407+0000", 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1643e-05 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.058511+0000", 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30110401399999998 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.058791+0000", 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027993799999999998 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.362316+0000", 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.303525292 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.362488+0000", 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000171693 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:53.563839+0000", 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20135165499999999 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:972 3.cs0 3:372a9655:::smithi16530038-32:head [append 0~720896] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:55.659190+0000", 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.537593589, 2024-09-16T04:43:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.695971122, 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 972 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.659190+0000", 2024-09-16T04:43:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.659190+0000", 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.659192+0000", 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4870000000000001e-06 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.659832+0000", 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064000299999999999 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.659854+0000", 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1764e-05 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.659868+0000", 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3694e-05 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.744912+0000", 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.085044146000000001 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.744924+0000", 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.206e-05 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.748494+0000", 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003569563 2024-09-16T04:43:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.748682+0000", 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000188078 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.750471+0000", 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017894199999999999 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.754007+0000", 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035363149999999999 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.354921+0000", 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60091321399999997 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.355161+0000", 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024037799999999999 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:973 3.cs0 3:372a9655:::smithi16530038-32:head [append 0~540672] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:55.666426+0000", 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.53035802, 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.68993053400000004, 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 973 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.666426+0000", 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.666426+0000", 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.666427+0000", 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.178e-06 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.666756+0000", 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032908600000000003 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.666773+0000", 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7002000000000001e-05 2024-09-16T04:43:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.666784+0000", 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1429e-05 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.752016+0000", 2024-09-16T04:43:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.085231985999999996 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.752158+0000", 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141741 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.753136+0000", 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00097784299999999994 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.755921+0000", 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027852089999999999 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.356135+0000", 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60021360800000001 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.356356+0000", 2024-09-16T04:43:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022145199999999999 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:974 3.cs0 3:372a9655:::smithi16530038-32:head [append 0~688128] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:55.675764+0000", 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.52101976, 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.68128830500000004, 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 974 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.675764+0000", 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.675764+0000", 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.675766+0000", 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4080000000000001e-06 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676451+0000", 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068477299999999996 2024-09-16T04:43:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676474+0000", 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3042000000000001e-05 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676488+0000", 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4496e-05 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.754150+0000", 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.077661163000000005 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.754288+0000", 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000138722 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.755288+0000", 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0009995920000000001 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.757907+0000", 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002619447 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.356932+0000", 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59902422899999996 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.357052+0000", 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120433 2024-09-16T04:43:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:976 3.cs0 3:372a9655:::smithi16530038-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:55.676551+0000", 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.520233008, 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88447735500000002, 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 976 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676551+0000", 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676551+0000", 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676551+0000", 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5199999999999995e-07 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676555+0000", 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.676571+0000", 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6358000000000002e-05 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.776673+0000", 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100101536 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.852144+0000", 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.075471269999999993 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.852473+0000", 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032889399999999999 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.357593+0000", 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50512003699999997 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.357759+0000", 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016566899999999999 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.561028+0000", 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20326923099999999 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:06.327 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:43:06.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:43:06.329 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:06.322+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:06.430 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:43:06.422+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:43:06.467 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:06.467 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:06.467 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:43:06.467 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:06.481 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:43:06.530 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:06.526+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:06.631 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:43:06.626+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:43:06.657 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:06.657 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:06.657 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:43:06.657 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:43:06.657 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:06.671 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:43:06.758 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:06.754+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:06.833 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:06.830+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:06.847 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:164 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.945293616000001, 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40676020400000001, 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:06.848 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 164 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897534+0000", 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897538+0000", 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0500000000000002e-06 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897554+0000", 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5886e-05 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.850 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897591+0000", 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6899e-05 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897617+0000", 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5922e-05 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897727+0000", 2024-09-16T04:43:06.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011011200000000001 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897835+0000", 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107889 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304263+0000", 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40642834999999999 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304295+0000", 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1096000000000002e-05 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:165 3.6s0 3:6877e850:::smithi16530038-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.945153927999996, 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20425574899999999, 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.853 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 165 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897674+0000", 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897676+0000", 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5030000000000001e-06 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897685+0000", 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.183e-06 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.855 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.897709+0000", 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4216e-05 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997840+0000", 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100131417 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098015+0000", 2024-09-16T04:43:06.856 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100174894 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098124+0000", 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108714 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101885+0000", 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037611459999999999 2024-09-16T04:43:06.857 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.101930+0000", 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4675999999999997e-05 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:166 3.6s0 3.a17ee16 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.844937665000003, 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.306539124, 2024-09-16T04:43:06.858 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 166 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997890+0000", 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997892+0000", 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0899999999999999e-06 2024-09-16T04:43:06.860 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997905+0000", 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2583e-05 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.997926+0000", 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.101e-05 2024-09-16T04:43:06.861 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.098022+0000", 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100096478 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304408+0000", 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20638583099999999 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.862 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:47.304429+0000", 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1132e-05 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.16s0 42/41 ECSubRead(tid=6, to_read={3:6877e850:::smithi16530038-5:head=0,20480,0}, subchunks={3:6877e850:::smithi16530038-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 74.534604599000005, 2024-09-16T04:43:06.863 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20348143299999999, 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:43:06.864 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308223+0000", 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.865 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308260+0000", 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6318e-05 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308312+0000", 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2611999999999997e-05 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511485+0000", 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20317243800000001 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.866 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511705+0000", 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220065 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 74.533928766000002, 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20303517200000001, 2024-09-16T04:43:06.867 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308899+0000", 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.868 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308901+0000", 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.308970+0000", 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9240999999999997e-05 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309005+0000", 2024-09-16T04:43:06.869 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4314999999999999e-05 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309023+0000", 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8879999999999999e-05 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511826+0000", 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202802133 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.870 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.511934+0000", 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108756 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.16s0 42/41 ECSubReadReply(tid=6, attrs_read=0))", 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 74.533693974000002, 2024-09-16T04:43:06.871 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20302847199999999, 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309134+0000", 2024-09-16T04:43:06.872 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309135+0000", 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.39e-06 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309242+0000", 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106959 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.873 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309260+0000", 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7197000000000002e-05 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.309270+0000", 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0339e-05 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512036+0000", 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20276591199999999 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.874 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.512162+0000", 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126675 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.14s4 43/41 ECSubWrite(tid=30, reqid=client.4728.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=42'15))", 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 73.503551572000006, 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20195774699999999, 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339276+0000", 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.876 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339277+0000", 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.61e-07 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339289+0000", 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1533e-05 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.339303+0000", 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4002000000000001e-05 2024-09-16T04:43:06.877 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439391+0000", 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10008851100000001 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439604+0000", 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301100000000001 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439617+0000", 2024-09-16T04:43:06.878 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3076e-05 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541138+0000", 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10152072299999999 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.541234+0000", 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6130000000000003e-05 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.879 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 53/32 ECSubWrite(tid=89, reqid=client.4728.0:426, at_version=53'29, trim_to=0'0, roll_forward_to=53'28))", 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 62.908633555000002, 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20199651099999999, 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.880 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934194+0000", 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.881 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934195+0000", 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934336+0000", 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014033399999999999 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:03.934357+0000", 2024-09-16T04:43:06.882 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1769000000000001e-05 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034519+0000", 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016164599999999 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034685+0000", 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000166312 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.883 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.034712+0000", 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6959999999999999e-05 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136138+0000", 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101425772 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:04.136191+0000", 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2868000000000003e-05 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=115, reqid=client.4728.0:457, at_version=58'26, trim_to=0'0, roll_forward_to=57'24))", 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 58.543472395999999, 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.240164188, 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299356+0000", 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5699999999999999e-07 2024-09-16T04:43:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299490+0000", 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013428299999999999 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299510+0000", 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437999999999999e-05 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.299523+0000", 2024-09-16T04:43:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3132000000000001e-05 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436652+0000", 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.13712892300000001 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.436659+0000", 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4410000000000004e-06 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539460+0000", 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.102800512 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.539520+0000", 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0201999999999997e-05 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 58/32 ECSubWrite(tid=122, reqid=client.4728.0:460, at_version=0'0, trim_to=0'0, roll_forward_to=58'29))", 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 58.030346584999997, 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20212734700000001, 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812481+0000", 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812484+0000", 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3700000000000002e-06 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812511+0000", 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.775e-05 2024-09-16T04:43:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812550+0000", 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812567+0000", 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7186000000000001e-05 2024-09-16T04:43:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812770+0000", 2024-09-16T04:43:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202862 2024-09-16T04:43:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.812799+0000", 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8716000000000001e-05 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014571+0000", 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.201771586 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.014609+0000", 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8124999999999998e-05 2024-09-16T04:43:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 51.590034817000003, 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.296481516, 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252793+0000", 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.252794+0000", 2024-09-16T04:43:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9699999999999996e-07 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253432+0000", 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063874400000000001 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253463+0000", 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0052999999999998e-05 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.253487+0000", 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4335999999999999e-05 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344896+0000", 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.091408780999999995 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.344925+0000", 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9490000000000001e-05 2024-09-16T04:43:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549182+0000", 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.204257051 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549275+0000", 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2464000000000001e-05 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 51.584243960999999, 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29077158600000003, 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258584+0000", 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258586+0000", 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6169999999999999e-06 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258848+0000", 2024-09-16T04:43:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026226500000000002 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258888+0000", 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0395999999999997e-05 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.258917+0000", 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8371999999999999e-05 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.346992+0000", 2024-09-16T04:43:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.088075353999999995 2024-09-16T04:43:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.347031+0000", 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8751999999999998e-05 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549288+0000", 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20225760800000001 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549356+0000", 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7222000000000001e-05 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 51.581377226999997, 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.288032389, 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261451+0000", 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261453+0000", 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.79e-06 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261566+0000", 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000113801 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261597+0000", 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0772999999999998e-05 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.261613+0000", 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5939000000000001e-05 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348618+0000", 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.087004695000000007 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.348634+0000", 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6246e-05 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549401+0000", 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.200766994 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549483+0000", 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2150999999999993e-05 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 51.579682206999998, 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28676506400000001, 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263146+0000", 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263148+0000", 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.81e-06 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263180+0000", 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2019e-05 2024-09-16T04:43:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263210+0000", 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0429999999999998e-05 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.263226+0000", 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6093e-05 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548818+0000", 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28559150300000002 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.548834+0000", 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6804e-05 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549811+0000", 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00097657100000000008 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:15.549911+0000", 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9834000000000007e-05 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/67 ECSubWrite(tid=159, reqid=client.4728.0:622, at_version=74'134, trim_to=0'0, roll_forward_to=72'132))", 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.369231480000003, 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.205798497, 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473596+0000", 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473597+0000", 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3999999999999998e-07 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473620+0000", 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3383999999999998e-05 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473630+0000", 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.473639+0000", 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9799999999999993e-06 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676469+0000", 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.202829168 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.676503+0000", 2024-09-16T04:43:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4239999999999997e-05 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679328+0000", 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028252860000000002 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.679395+0000", 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6773000000000006e-05 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDScrubReserve(3.21s5,QREQUEST,e:74,from:2(0),reservation_nonce:2)", 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 41.906262542, 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30070015300000003, 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:43:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936565+0000", 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936567+0000", 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.86e-06 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936576+0000", 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0559999999999994e-06 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.936605+0000", 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8632999999999998e-05 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.036785+0000", 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10017992000000001 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237143+0000", 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20035802999999999 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237193+0000", 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0121999999999999e-05 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.237266+0000", 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2532000000000003e-05 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.bs1 76/67 ECSubRead(tid=164, to_read={3:d388eed4:::smithi16530038-28:head=0,339968,0}, subchunks={3:d388eed4:::smithi16530038-28:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.564921400999999, 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22646649499999999, 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277907+0000", 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277908+0000", 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7439999999999999e-06 2024-09-16T04:43:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277925+0000", 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7084000000000001e-05 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277960+0000", 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4208999999999997e-05 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.277978+0000", 2024-09-16T04:43:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8219e-05 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504200+0000", 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22622223899999999 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.504373+0000", 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000173 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 90/37 ECSubWrite(tid=82, reqid=osd.15.0:82, at_version=90'74, trim_to=0'0, roll_forward_to=88'70))", 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:40.482601+0000", 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.360227116000001, 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20413171399999999, 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482601+0000", 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482601+0000", 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482603+0000", 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1629999999999999e-06 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482659+0000", 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6325e-05 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482690+0000", 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0442000000000001e-05 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.482713+0000", 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3238999999999999e-05 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.584033+0000", 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101320049 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.584054+0000", 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0679e-05 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.686637+0000", 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10258316100000001 2024-09-16T04:43:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:40.686733+0000", 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5655999999999994e-05 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.1fs6 102/97 ECSubWrite(tid=276, reqid=osd.11.0:276, at_version=102'62, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:52.909577+0000", 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 13.9332507, 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20182298400000001, 2024-09-16T04:43:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.909577+0000", 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.909577+0000", 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.909579+0000", 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.61e-06 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.909620+0000", 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1081999999999997e-05 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.909648+0000", 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8523000000000001e-05 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:53.009782+0000", 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100133465 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:53.110010+0000", 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10022846000000001 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:53.110027+0000", 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6135999999999999e-05 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:53.111323+0000", 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00129634 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:53.111400+0000", 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7367999999999995e-05 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs2 105/67 ECSubWrite(tid=288, reqid=client.4728.0:973, at_version=105'109, trim_to=0'0, roll_forward_to=102'105))", 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:55.850815+0000", 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.992013125, 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20194061899999999, 2024-09-16T04:43:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.850815+0000", 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.850815+0000", 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.850817+0000", 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7400000000000001e-06 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.851016+0000", 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019899199999999999 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.851048+0000", 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2449000000000001e-05 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.951161+0000", 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100113016 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.951549+0000", 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00038844699999999999 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.951581+0000", 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1809e-05 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.052694+0000", 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101113178 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.052755+0000", 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0988e-05 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:06.949 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:06.950 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:43:06.907+0000 7fb4cc2707c0 -1 Falling back to public interface 2024-09-16T04:43:06.951 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:43:06.952 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:43:06.947+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:43:07.051 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:07.046+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:07.128 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:07.128 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:07.128 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:43:07.128 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:07.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:43:07.152 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:43:07.151+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:43:07.253 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:07.246+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:07.314 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:07.314 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:07.314 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:43:07.314 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:43:07.314 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:07.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:43:07.354 INFO:tasks.ceph.osd.11.smithi165.stderr:2024-09-16T04:43:07.351+0000 7f499be91640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26663) UID: 0 2024-09-16T04:43:07.455 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:07.451+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:07.503 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:07.503 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:43:07.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:43:07.503 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:137 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 81.436313318000003, 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50711364199999998, 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.504 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 137 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062313+0000", 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.505 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062314+0000", 2024-09-16T04:43:07.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:43:07.509 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.509 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.509 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.509 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062318+0000", 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1710000000000004e-06 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.062327+0000", 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0089999999999996e-06 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162444+0000", 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10011658 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163173+0000", 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00072917800000000003 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.163440+0000", 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026697899999999998 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164004+0000", 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056357499999999997 2024-09-16T04:43:07.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.164775+0000", 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00077136499999999998 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569249+0000", 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.404473996 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.512 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.569427+0000", 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177906 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:138 3.1s0 3:8c8bcb2a:::smithi16530038-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 81.336144434999994, 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.71018989099999996, 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.513 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 138 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.514 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162482+0000", 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162484+0000", 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162491+0000", 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162507+0000", 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5141e-05 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.162514+0000", 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8460000000000008e-06 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165370+0000", 2024-09-16T04:43:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028555719999999998 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.165438+0000", 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8466999999999993e-05 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570190+0000", 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40475178899999997 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.570271+0000", 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0437000000000003e-05 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.872672+0000", 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30240151100000001 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:181 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:07.518 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.406271168999993, 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.514197601, 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.519 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 181 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092356+0000", 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.092358+0000", 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9500000000000001e-06 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093295+0000", 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00093678900000000005 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093330+0000", 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5039000000000001e-05 2024-09-16T04:43:07.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.093348+0000", 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7210999999999998e-05 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.522 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269509+0000", 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17616195900000001 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.269804+0000", 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029419599999999999 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.272744+0000", 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029401789999999998 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279866+0000", 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007122499 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606426+0000", 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32655929 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606553+0000", 2024-09-16T04:43:07.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000127489 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:182 3.bs0 3:d388eed4:::smithi16530038-28:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.399942723999999, 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50813697599999996, 2024-09-16T04:43:07.525 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 182 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098684+0000", 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0260000000000002e-06 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099161+0000", 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000474556 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099192+0000", 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1612000000000002e-05 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099207+0000", 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4796e-05 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275225+0000", 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17601765799999999 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.275471+0000", 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024600099999999998 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.276908+0000", 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001437422 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280012+0000", 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031043899999999998 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606726+0000", 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.32671404399999998 2024-09-16T04:43:07.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.606821+0000", 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4470999999999997e-05 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:183 3.bs0 3:d388eed4:::smithi16530038-28:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.399392315, 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60643071599999998, 2024-09-16T04:43:07.531 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 183 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099234+0000", 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099235+0000", 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.017e-06 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099242+0000", 2024-09-16T04:43:07.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4230000000000002e-06 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099256+0000", 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3936e-05 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.099264+0000", 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278076+0000", 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.17881208600000001 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.278318+0000", 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024238100000000001 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.279368+0000", 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010492209999999999 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.535 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281553+0000", 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021853739999999999 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705572+0000", 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.42401889100000001 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.705665+0000", 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3173999999999999e-05 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.536 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:191 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.397532159999997, 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68460226999999996, 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.537 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 191 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101095+0000", 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101096+0000", 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-06 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101104+0000", 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1240000000000005e-06 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101122+0000", 2024-09-16T04:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8045000000000001e-05 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101130+0000", 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2670000000000006e-06 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280859+0000", 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179728363 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.280864+0000", 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7370000000000003e-06 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.405994+0000", 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.12512986500000001 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.406243+0000", 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024852700000000001 2024-09-16T04:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.785697+0000", 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.37945393900000002 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:192 3.1s0 3:8583b9f2:::smithi16530038-39:2 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.397479942000004, 2024-09-16T04:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68518017200000003, 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 192 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101147+0000", 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0299999999999999e-07 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101154+0000", 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3300000000000004e-06 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101160+0000", 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.348e-06 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101164+0000", 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2899999999999996e-06 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281071+0000", 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.179906749 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.281074+0000", 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.976e-06 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506776+0000", 2024-09-16T04:43:07.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22570168299999999 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.506893+0000", 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011752300000000001 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.786327+0000", 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27943377000000003 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.547 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:201 3.bs0 3:d797f264:::smithi16530038-12:2 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 79.397346854999995, 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.68324074999999995, 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 201 2024-09-16T04:43:07.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101280+0000", 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101282+0000", 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9750000000000001e-06 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.101287+0000", 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.651e-06 2024-09-16T04:43:07.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.201419+0000", 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10013264099999999 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482351+0000", 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28093166899999999 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.482482+0000", 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013094199999999999 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:48.784521+0000", 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30203853899999999 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.552 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:372 3.1s0 3:8583b9f2:::smithi16530038-39:3 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 68.317344556999998, 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.58438020099999999, 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 372 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181282+0000", 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181283+0000", 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1100000000000005e-07 2024-09-16T04:43:07.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181288+0000", 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4739999999999996e-06 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181300+0000", 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1266e-05 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181308+0000", 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5669999999999995e-06 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181619+0000", 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031025600000000002 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.181627+0000", 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3329999999999997e-06 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288322+0000", 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669511499999999 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.288524+0000", 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000202511 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.765662+0000", 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.47713786800000002 2024-09-16T04:43:07.557 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:452 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.239672701000003, 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57389822599999996, 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 452 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258954+0000", 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.258957+0000", 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9510000000000002e-06 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260061+0000", 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011037410000000001 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260099+0000", 2024-09-16T04:43:07.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8401e-05 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260123+0000", 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3808999999999999e-05 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260226+0000", 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010314000000000001 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.260310+0000", 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3652000000000005e-05 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.261217+0000", 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090691199999999995 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.262686+0000", 2024-09-16T04:43:07.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014689 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832721+0000", 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.57003547600000004 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.832852+0000", 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013124400000000001 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:453 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~622592] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.228994827000001, 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56351327799999995, 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 453 2024-09-16T04:43:07.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269632+0000", 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.269634+0000", 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2069999999999998e-06 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270168+0000", 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053375500000000002 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270185+0000", 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7231000000000002e-05 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270194+0000", 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4349999999999997e-06 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270266+0000", 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2702999999999998e-05 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.270366+0000", 2024-09-16T04:43:07.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9485e-05 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271233+0000", 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00086726900000000005 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374033+0000", 2024-09-16T04:43:07.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10280007300000001 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833038+0000", 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45900526899999999 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.833145+0000", 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106851 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.569 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:454 3.1s0 3:803446a7:::smithi16530038-31:head [append 0~147456] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.226730046999997, 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56224230100000006, 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.570 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 454 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271897+0000", 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.271898+0000", 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.364e-06 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272075+0000", 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017664100000000001 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272088+0000", 2024-09-16T04:43:07.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3192e-05 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272097+0000", 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4720000000000001e-06 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372245+0000", 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10014781 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372332+0000", 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7258999999999999e-05 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.372854+0000", 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052174600000000002 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374728+0000", 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018736219999999999 2024-09-16T04:43:07.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834034+0000", 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.45930626099999999 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.834139+0000", 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000104934 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:455 3.1s0 3:803446a7:::smithi16530038-31:head [setxattr _header (57) in=7b] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.226505684000003, 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.66155386500000002, 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 455 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.576 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272121+0000", 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272122+0000", 2024-09-16T04:43:07.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272127+0000", 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5369999999999997e-06 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272135+0000", 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272141+0000", 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5999999999999997e-06 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373376+0000", 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.101234779 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373445+0000", 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9507999999999999e-05 2024-09-16T04:43:07.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373742+0000", 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029695500000000001 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374731+0000", 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098912400000000008 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933530+0000", 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.55879828600000003 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.933675+0000", 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145353 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:456 3.1s0 3:803446a7:::smithi16530038-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:43:07.581 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.226474500999998, 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.76406687699999998, 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 456 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272152+0000", 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272153+0000", 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0100000000000005e-07 2024-09-16T04:43:07.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272155+0000", 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.638e-06 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272165+0000", 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1579999999999994e-06 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.272171+0000", 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.373963+0000", 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10179178799999999 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.374024+0000", 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0791999999999997e-05 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934462+0000", 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.56043857600000002 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.934706+0000", 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024387000000000001 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.036219+0000", 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10151300000000001 2024-09-16T04:43:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:458 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~638976] snapc d=[d,c,b,a,9,7] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.195205436999998, 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50818626099999997, 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.587 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 458 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303421+0000", 2024-09-16T04:43:07.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303426+0000", 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5870000000000003e-06 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303961+0000", 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00053495599999999995 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.303975+0000", 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3835999999999999e-05 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404043+0000", 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100068845 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404113+0000", 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9350999999999994e-05 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404242+0000", 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012877400000000001 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.405312+0000", 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001070812 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.407819+0000", 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002506131 2024-09-16T04:43:07.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811485+0000", 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40366653800000002 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.811608+0000", 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012243099999999999 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:461 3.bs0 3:d1897791:::smithi16530038-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 59.094472734999997, 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.61182177000000004, 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 461 2024-09-16T04:43:07.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404154+0000", 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6399999999999998e-07 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404155+0000", 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2550000000000001e-06 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404160+0000", 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0890000000000002e-06 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.404162+0000", 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8399999999999999e-06 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.507951+0000", 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103788373 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.508127+0000", 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017614900000000001 2024-09-16T04:43:07.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813061+0000", 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30493453100000001 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:08.813333+0000", 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027161699999999998 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:09.015976+0000", 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20264265200000001 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:574 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 48.194863748000003, 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50935247100000003, 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.598 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 574 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:43:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303763+0000", 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303765+0000", 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.827e-06 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303772+0000", 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6210000000000004e-06 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.303788+0000", 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5760000000000002e-05 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.403903+0000", 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100114464 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404144+0000", 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024159600000000001 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404641+0000", 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000496598 2024-09-16T04:43:07.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.406835+0000", 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021939149999999998 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511811+0000", 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.104976429 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.812951+0000", 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30113980800000001 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.813115+0000", 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016445300000000001 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:576 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-16T04:43:07.604 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 48.094598083000001, 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51026414200000003, 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 576 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404029+0000", 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6199999999999998e-07 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404032+0000", 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6280000000000001e-06 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404038+0000", 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6200000000000001e-06 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.404043+0000", 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5020000000000003e-06 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.510735+0000", 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10669203400000001 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.510837+0000", 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101847 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.511366+0000", 2024-09-16T04:43:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052975600000000004 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.512713+0000", 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001346168 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.914111+0000", 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40139793200000001 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:19.914293+0000", 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018219300000000001 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:644 3.bs0 3:d388eed4:::smithi16530038-28:head [read 0~1359872,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 41.321316043000003, 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51132577199999996, 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:07.610 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 644 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:43:07.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177311+0000", 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7e-07 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177315+0000", 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.523e-06 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177322+0000", 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.968e-06 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.177327+0000", 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.631e-06 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.458181+0000", 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.28085389999999999 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.458352+0000", 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017090399999999999 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:26.688636+0000", 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.23028437600000001 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.614 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:717 3.bs0 3:d1897791:::smithi16530038-16:head [append 0~802816] snapc 16=[16,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.050331589000002, 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50523997499999995, 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 717 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:07.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448295+0000", 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.448297+0000", 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2539999999999999e-06 2024-09-16T04:43:07.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449278+0000", 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098078800000000002 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449311+0000", 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2768999999999997e-05 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.449326+0000", 2024-09-16T04:43:07.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5392000000000001e-05 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.537454+0000", 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.088127592000000005 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.537466+0000", 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2189e-05 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:07.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.544288+0000", 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0068214950000000003 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.544574+0000", 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000286113 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.547183+0000", 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026089809999999998 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.651536+0000", 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.104353801 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.953403+0000", 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30186636999999999 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:31.953535+0000", 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013223099999999999 2024-09-16T04:43:07.620 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.621 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.621 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.621 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:07.621 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:07.621 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:07.622 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:43:07.623 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:07.618+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:07.697 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:43:07.742 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:43:07.738+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:43:07.757 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:07.757 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:07.757 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:43:07.757 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:07.773 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:43:07.823 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:07.822+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:07.924 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:07.922+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:07.934 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:43:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:43:07.934 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:07.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:43:08.025 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:08.022+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:08.074 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:08.074 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-09-16T04:43:08.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-09-16T04:43:08.074 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-09-16T04:43:08.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.074 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:148 3.4s0 3:2b4d2b94:::smithi16530038-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:08.074 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 81.871473395999999, 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.49811553600000003, 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 148 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198239+0000", 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198240+0000", 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3600000000000003e-07 2024-09-16T04:43:08.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198243+0000", 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.281e-06 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.198250+0000", 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7870000000000004e-06 2024-09-16T04:43:08.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298324+0000", 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100073891 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298376+0000", 2024-09-16T04:43:08.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2790000000000001e-05 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298437+0000", 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0550999999999997e-05 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298723+0000", 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000285741 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.299721+0000", 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00099841899999999996 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696249+0000", 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.39652776099999998 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696354+0000", 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105679 2024-09-16T04:43:08.080 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:149 3.4s0 3:2b4d2b94:::smithi16530038-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 81.771360917999999, 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.59985384399999997, 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.081 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.082 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.082 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 149 2024-09-16T04:43:08.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.082 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:43:08.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298351+0000", 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298354+0000", 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298362+0000", 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3319999999999992e-06 2024-09-16T04:43:08.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298377+0000", 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5387999999999999e-05 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.298388+0000", 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0957e-05 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398555+0000", 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10016630999999999 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.398706+0000", 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015114699999999999 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696493+0000", 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.29778684 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.696643+0000", 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015093699999999999 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.898205+0000", 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20156174700000001 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:160 3.4s0 3:2d731f27:::smithi16530038-30:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 81.615730851999999, 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44291274600000002, 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 160 2024-09-16T04:43:08.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453981+0000", 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453983+0000", 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4729999999999999e-06 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.453989+0000", 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1299999999999998e-06 2024-09-16T04:43:08.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454005+0000", 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5880999999999999e-05 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454014+0000", 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1079999999999999e-06 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454090+0000", 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6062999999999993e-05 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454166+0000", 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5772999999999994e-05 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.454458+0000", 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029229599999999999 2024-09-16T04:43:08.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.455106+0000", 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00064800899999999995 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896766+0000", 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44165951199999998 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:46.896894+0000", 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000128501 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.14s0 42/41 ECSubReadReply(tid=27, attrs_read=0))", 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 75.237012136000004, 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60655789999999998, 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.094 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832700+0000", 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832701+0000", 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6400000000000001e-07 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832733+0000", 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2382000000000003e-05 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.832757+0000", 2024-09-16T04:43:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3691999999999999e-05 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:52.932887+0000", 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100130238 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.438947+0000", 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50605998399999996 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:53.439258+0000", 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031074 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:368 3.14s0 3:28a14731:::smithi16530038-3:head [setxattr _header (55) in=7b] snapc 6=[3] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 68.884671443000002, 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50583271100000005, 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.098 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 368 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185041+0000", 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185043+0000", 2024-09-16T04:43:08.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.463e-06 2024-09-16T04:43:08.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185054+0000", 2024-09-16T04:43:08.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0356e-05 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185080+0000", 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6519000000000001e-05 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185097+0000", 2024-09-16T04:43:08.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.717e-05 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.188751+0000", 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036535249999999999 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.189019+0000", 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026812599999999999 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.190416+0000", 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013973830000000001 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.389174+0000", 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19875738700000001 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690722+0000", 2024-09-16T04:43:08.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30154792000000002 2024-09-16T04:43:08.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.690874+0000", 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000151862 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:369 3.14s0 3:28a14731:::smithi16530038-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 68.884589726000002, 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.81195188399999996, 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.107 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 369 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185123+0000", 2024-09-16T04:43:08.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185124+0000", 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.33e-06 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185130+0000", 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9070000000000004e-06 2024-09-16T04:43:08.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185154+0000", 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4059000000000001e-05 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.185173+0000", 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8936e-05 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388708+0000", 2024-09-16T04:43:08.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.203534774 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.388994+0000", 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028647200000000002 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691636+0000", 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.302641681 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.691904+0000", 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026824200000000002 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:41:59.997074+0000", 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30517048299999999 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.113 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:618 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [append 0~0] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.700413165999997, 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.54505251200000004, 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 618 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369299+0000", 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369302+0000", 2024-09-16T04:43:08.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6639999999999998e-06 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369315+0000", 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3416e-05 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369345+0000", 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9961e-05 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369359+0000", 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4107000000000001e-05 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.604180+0000", 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.23482078300000001 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.604381+0000", 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020054599999999999 2024-09-16T04:43:08.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.605620+0000", 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012394699999999999 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.608601+0000", 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029811059999999999 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914102+0000", 2024-09-16T04:43:08.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.305500773 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914352+0000", 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024968599999999999 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:619 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [rollback 11] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:43:08.119 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.700325657, 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.54536771900000003, 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 619 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369387+0000", 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.68e-07 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369390+0000", 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.948e-06 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369399+0000", 2024-09-16T04:43:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3499999999999997e-06 2024-09-16T04:43:08.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369405+0000", 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.455e-06 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.606687+0000", 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.237281837 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.606930+0000", 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024356099999999999 2024-09-16T04:43:08.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.607700+0000", 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000769507 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.810765+0000", 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.20306539400000001 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914602+0000", 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103836502 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914754+0000", 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000152597 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:620 3.1es0 3:7ee6b8f9:::smithi16530038-8:head [append 0~0] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:08.128 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.700299899000001, 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.81742312399999995, 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 620 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369412+0000", 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369413+0000", 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9099999999999999e-07 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369426+0000", 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.365e-05 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.369435+0000", 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4700000000000002e-06 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.469519+0000", 2024-09-16T04:43:08.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.100084346 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.912089+0000", 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.44256948200000001 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.912270+0000", 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018106700000000001 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.913255+0000", 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098478400000000005 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:24.914789+0000", 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015345210000000001 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.186759+0000", 2024-09-16T04:43:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.27197024199999997 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:25.186836+0000", 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6171e-05 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:707 3.1es0 3:78f34faf:::smithi16530038-13:head [rollback 11] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:43:08.134 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.207679433000003, 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51046276300000004, 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 707 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862033+0000", 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862034+0000", 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862038+0000", 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9160000000000003e-06 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862049+0000", 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1786e-05 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862057+0000", 2024-09-16T04:43:08.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0669999999999992e-06 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.966992+0000", 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10493451600000001 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.967212+0000", 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021991 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.968498+0000", 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012864350000000001 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.271112+0000", 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30261409299999997 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372370+0000", 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10125774899999999 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.372496+0000", 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000125501 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:708 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.207598339, 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.74368462499999999, 2024-09-16T04:43:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 708 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862114+0000", 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7800000000000002e-07 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862116+0000", 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.08e-06 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862122+0000", 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7930000000000003e-06 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:29.862126+0000", 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2470000000000002e-06 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373611+0000", 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.51148433299999996 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.373943+0000", 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00033231500000000002 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.375178+0000", 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001235367 2024-09-16T04:43:08.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.376541+0000", 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013628410000000001 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.605718+0000", 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.22917667899999999 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:30.605799+0000", 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0592000000000004e-05 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 92/67 ECSubWrite(tid=156, reqid=osd.2.0:156, at_version=92'88, trim_to=0'0, roll_forward_to=90'84))", 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:42.515434+0000", 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.554278487000001, 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.49799755800000001, 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:08.146 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515434+0000", 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515434+0000", 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.147 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515435+0000", 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1629999999999999e-06 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515466+0000", 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0927999999999999e-05 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.515487+0000", 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0792e-05 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.148 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:42.615560+0000", 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10007328 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.011802+0000", 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.396242135 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.011814+0000", 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.216e-05 2024-09-16T04:43:08.149 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.013372+0000", 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015577449999999999 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:43.013431+0000", 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9354999999999997e-05 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:943 3.5s1 3.2115b2a5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:51.580647+0000", 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 16.489065315000001, 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0139504159999999, 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 943 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.151 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580647+0000", 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580647+0000", 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.152 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580651+0000", 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9249999999999997e-06 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580668+0000", 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6949999999999999e-05 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580699+0000", 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1359999999999998e-05 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.153 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580716+0000", 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6905000000000002e-05 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580807+0000", 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0600999999999995e-05 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0634e-05 2024-09-16T04:43:08.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.584336+0000", 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003518916 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.584341+0000", 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8919999999999999e-06 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.594585+0000", 2024-09-16T04:43:08.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0102435540000001 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.594597+0000", 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2679e-05 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:944 3.5s1 3.2115b2a5 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:51.580752+0000", 2024-09-16T04:43:08.156 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 16.488959892, 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0138833140000001, 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 944 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580752+0000", 2024-09-16T04:43:08.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580752+0000", 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580753+0000", 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2200000000000003e-07 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580757+0000", 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9519999999999996e-06 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580768+0000", 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1189000000000001e-05 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580776+0000", 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6729999999999992e-06 2024-09-16T04:43:08.159 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580919+0000", 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014272300000000001 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.580927+0000", 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.4440000000000008e-06 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.684488+0000", 2024-09-16T04:43:08.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103560847 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for peered", 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:51.684493+0000", 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.904e-06 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.594629+0000", 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.91013564400000002 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:52.594636+0000", 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.216e-06 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:982 3.4s0 3:208a5e75:::smithi16530038-37:head [append 0~475136] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:55.740900+0000", 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.328811925, 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50723840799999997, 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.162 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 982 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.740900+0000", 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.740900+0000", 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.740903+0000", 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.48e-06 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.741789+0000", 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00088578400000000003 2024-09-16T04:43:08.164 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.741823+0000", 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4202999999999998e-05 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.841919+0000", 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.10009643999999999 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.923642+0000", 2024-09-16T04:43:08.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.081722443000000006 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.923847+0000", 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020523500000000001 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.925307+0000", 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001459541 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.929624+0000", 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043171299999999998 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.248001+0000", 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.31837776800000001 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.248139+0000", 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000137384 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.167 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:983 3.4s0 3:208a5e75:::smithi16530038-37:head [append 0~655360] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:55.841978+0000", 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.227734069, 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.606625584, 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.168 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 983 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.841978+0000", 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.841978+0000", 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.169 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.841980+0000", 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9489999999999999e-06 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.843059+0000", 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001078374 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.843176+0000", 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000117213 2024-09-16T04:43:08.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.843194+0000", 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7790000000000001e-05 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.926813+0000", 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.083619763 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.926971+0000", 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015735599999999999 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.928512+0000", 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015414420000000001 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.033010+0000", 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.104497721 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.448462+0000", 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.41545216299999999 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.448604+0000", 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014181299999999999 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.173 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:984 3.4s0 3:208a5e75:::smithi16530038-37:head [append 0~671744] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:55.843217+0000", 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.226495177, 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.60586421400000001, 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 984 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.174 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.843217+0000", 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.843217+0000", 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.843218+0000", 2024-09-16T04:43:08.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2100000000000001e-06 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946509+0000", 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103290861 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946549+0000", 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9906e-05 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946572+0000", 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2410000000000001e-05 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.044772+0000", 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.098200781000000001 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.044960+0000", 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018763499999999999 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.046589+0000", 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016286549999999999 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.136706+0000", 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.090117199999999995 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.448965+0000", 2024-09-16T04:43:08.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.312259026 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.449081+0000", 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011653 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:985 3.4s0 3:208a5e75:::smithi16530038-37:head [append 0~147456] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:55.946607+0000", 2024-09-16T04:43:08.179 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.123105718, 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.50275918200000003, 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 985 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946607+0000", 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946607+0000", 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946609+0000", 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4269999999999998e-06 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946927+0000", 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031841500000000001 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946962+0000", 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5036000000000001e-05 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.946983+0000", 2024-09-16T04:43:08.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0499000000000001e-05 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.137179+0000", 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19019561700000001 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.137357+0000", 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017845900000000001 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.138455+0000", 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001097797 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.141247+0000", 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027925039999999999 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.449257+0000", 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30800956099999999 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.449366+0000", 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108867 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:986 3.4s0 3:208a5e75:::smithi16530038-37:head [setxattr _header (57) in=7b] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:55.947013+0000", 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.122699520999999, 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.503042459, 2024-09-16T04:43:08.185 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 986 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947013+0000", 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947013+0000", 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947014+0000", 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5960000000000001e-06 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947022+0000", 2024-09-16T04:43:08.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4869999999999996e-06 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947043+0000", 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1608999999999999e-05 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947059+0000", 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5953000000000001e-05 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.139276+0000", 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.192216213 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.139451+0000", 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017552900000000001 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.140305+0000", 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00085349000000000004 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.142578+0000", 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0022734360000000002 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.449765+0000", 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.30718716299999999 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.450055+0000", 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028998300000000001 2024-09-16T04:43:08.190 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4728.0:987 3.4s0 3:208a5e75:::smithi16530038-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-09-16T04:42:55.947081+0000", 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.122630922999999, 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.70758097099999995, 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4728", 2024-09-16T04:43:08.191 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 987 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947081+0000", 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947081+0000", 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-09-16T04:43:08.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947083+0000", 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3230000000000001e-06 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947089+0000", 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3069999999999999e-06 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947109+0000", 2024-09-16T04:43:08.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0191999999999999e-05 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:55.947125+0000", 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.541e-05 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.144562+0000", 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.19743757300000001 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.144857+0000", 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00029443900000000002 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.551254+0000", 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.40639730200000002 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.551484+0000", 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000229958 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-09-16T04:43:08.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-09-16T04:42:56.654662+0000", 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.103178467 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-09-16T04:43:08.196 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:08.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T04:43:08.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:43:08.199 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:08.194+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:08.298 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:08.295+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:08.331 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:08.332 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:08.332 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-09-16T04:43:08.332 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:08.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T04:43:08.349 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:43:08.349 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:43:08.369 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:43:08.370 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:43:08.390 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:43:08.391 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:43:08.399 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:08.398+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:08.411 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:43:08.412 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:43:08.428 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:43:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:43:08.449 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:43:08.449 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:43:08.463 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:43:08.463 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-09-16T04:43:08.463 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-09-16T04:43:08.463 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-09-16T04:43:08.464 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:43:08.470 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:43:08.470 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:43:08.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T04:43:08.491 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:43:08.492 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:43:08.500 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:08.498+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:08.509 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:43:08.509 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:43:08.534 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:43:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:43:08.550 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:43:08.550 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:43:08.571 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:43:08.571 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:43:08.591 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:43:08.591 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:43:08.601 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:08.595+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:08.611 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:08.611 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:08.612 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:43:08.612 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:43:08.624 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:43:08.632 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:43:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:43:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.15 2024-09-16T04:43:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:43:08.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:43:08.693 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:43:08.700 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:08.694+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:08.758 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:08.759 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:08.773 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:43:08.801 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:08.799+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:08.893 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:08.893 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:08.904 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:08.898+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:08.907 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:43:09.005 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:08.998+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:09.031 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:09.031 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:09.046 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:43:09.106 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:43:09.103+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:43:09.172 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:09.172 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:09.187 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:43:09.207 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:09.202+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:09.305 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:09.305 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:09.308 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:43:09.306+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:43:09.321 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:43:09.409 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:09.403+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:09.487 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:09.488 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:09.502 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:43:09.510 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:09.507+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:09.611 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:09.606+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:09.631 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:09.631 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:09.647 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:43:09.711 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:09.707+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:09.771 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:09.772 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:09.787 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:43:09.831 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:09.830+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:09.909 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:09.909 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:09.912 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:09.910+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:09.924 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T04:43:10.015 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:10.014+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:10.091 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:10.091 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:10.106 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T04:43:10.118 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:10.115+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:10.216 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:10.210+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:10.272 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:10.273 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:10.287 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T04:43:10.331 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:10.330+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:10.410 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:10.410 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:10.418 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:10.410+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:10.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:43:10.518 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:43:10.515+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:43:10.557 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:10.557 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:10.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:43:10.619 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:10.618+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:10.636 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:43:10.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:43:10.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:43:10.739 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:10.739 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:10.753 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:43:10.762 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:43:10.758+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:43:10.797 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:43:10.797 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:43:10.820 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:10.815+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:10.833 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2024-09-16T04:43:10.833 INFO:teuthology.orchestra.run.smithi165.stderr:0 2024-09-16T04:43:10.833 INFO:teuthology.orchestra.run.smithi165.stderr:1 2024-09-16T04:43:10.833 INFO:teuthology.orchestra.run.smithi165.stderr:2 2024-09-16T04:43:10.833 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2024-09-16T04:43:10.833 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2024-09-16T04:43:10.834 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs debug_inject_read_zeros 2024-09-16T04:43:10.834 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs files list 2024-09-16T04:43:10.834 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs stats 2024-09-16T04:43:10.834 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator dump block 2024-09-16T04:43:10.834 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator fragmentation block 2024-09-16T04:43:10.834 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2024-09-16T04:43:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:43:10.836 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-15, ['dump_ops_in_flight'] 2024-09-16T04:43:10.836 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:43:10.837 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:43:10.860 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:43:10.860 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:43:10.861 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:43:10.859+0000 7fb4cc2707c0 -1 osd.15 99 log_to_monitors true 2024-09-16T04:43:10.881 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:43:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:108 3.es0 3:741a05b8:::smithi16530038-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 85.518149043999998, 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60644887800000002, 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.883 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 108 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.884 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362768+0000", 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362769+0000", 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.364e-06 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362776+0000", 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9789999999999996e-06 2024-09-16T04:43:10.885 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362795+0000", 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8569000000000001e-05 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.362811+0000", 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6084999999999999e-05 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.886 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369499+0000", 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066885180000000001 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.369745+0000", 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024616900000000002 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669288+0000", 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29954281100000002 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.887 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.669425+0000", 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013708200000000001 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:45.969217+0000", 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29979130100000001 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:507 3.es0 3:7777e846:::smithi16530038-17:head [copy-from ver 21] snapc 10=[10,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:43:10.888 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.609094106000001, 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.526543015, 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 507 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.889 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271823+0000", 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.890 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271824+0000", 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7990000000000001e-06 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271832+0000", 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.271852+0000", 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9194999999999998e-05 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.891 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.371991+0000", 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10013957800000001 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372175+0000", 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000183325 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.372198+0000", 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3306e-05 2024-09-16T04:43:10.892 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472740+0000", 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100542016 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.472902+0000", 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016220399999999999 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.893 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.692422+0000", 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21951984699999999 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.796322+0000", 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10389983699999999 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798260+0000", 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001938103 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.894 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:14.798366+0000", 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010582900000000001 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:582 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 51.541548050000003, 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69089629399999997, 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.895 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 582 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.896 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339369+0000", 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339370+0000", 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339374+0000", 2024-09-16T04:43:10.897 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0799999999999999e-06 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339387+0000", 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3464e-05 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.339414+0000", 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7223e-05 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.898 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446804+0000", 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.107389545 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.446957+0000", 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000153389 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727239+0000", 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28028220199999998 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.899 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.727462+0000", 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022226099999999999 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.030265+0000", 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280332199999999 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.900 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:613 3.es0 3:7777e846:::smithi16530038-17:head [append 0~589824] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.520697562000002, 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55445315100000003, 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.901 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 613 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360219+0000", 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.902 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.360223+0000", 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3730000000000001e-06 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361140+0000", 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00091743300000000005 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.903 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.904 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361178+0000", 2024-09-16T04:43:10.904 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7691999999999998e-05 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361197+0000", 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.891e-05 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361351+0000", 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015419300000000001 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:10.905 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.361371+0000", 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9726000000000001e-05 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366548+0000", 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051778730000000004 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.366714+0000", 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000165329 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.906 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.369378+0000", 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026643159999999999 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.372406+0000", 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003027891 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.907 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914520+0000", 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54211425499999999 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914672+0000", 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015216000000000001 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:614 3.es0 3:7777e846:::smithi16530038-17:head [append 0~540672] snapc 13=[13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:10.908 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.516724515999996, 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.55075342599999999, 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 614 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.909 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364192+0000", 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.364194+0000", 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.26e-06 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.910 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365041+0000", 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00084621499999999999 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.365073+0000", 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2719000000000003e-05 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465237+0000", 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100163671 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.911 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.912 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465392+0000", 2024-09-16T04:43:10.912 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015528300000000001 2024-09-16T04:43:10.912 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.912 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.912 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.912 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.465730+0000", 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000338005 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.467540+0000", 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001809583 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.469612+0000", 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020719760000000001 2024-09-16T04:43:10.913 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914847+0000", 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.44523504899999999 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.914 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.914946+0000", 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8665000000000005e-05 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:617 3.es0 3:7777e846:::smithi16530038-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.314210387000003, 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71993709900000002, 2024-09-16T04:43:10.915 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 617 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.916 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566706+0000", 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566707+0000", 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566713+0000", 2024-09-16T04:43:10.917 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.784e-06 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566725+0000", 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.173e-05 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566732+0000", 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6860000000000006e-06 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.918 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579439+0000", 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012706935000000001 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.579759+0000", 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032009600000000002 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074376+0000", 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49461618299999999 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.919 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.074520+0000", 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014481800000000001 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:25.286643+0000", 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212123017 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:696 3.10s0 3:0fc9379d:::smithi16530038-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:10.920 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.456208152999999, 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69507003199999995, 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 696 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424709+0000", 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.922 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424713+0000", 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5599999999999998e-06 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424724+0000", 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0906e-05 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.424732+0000", 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6499999999999996e-06 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.923 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502418+0000", 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077686693000000001 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.502530+0000", 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000111644 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917065+0000", 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41453470199999998 2024-09-16T04:43:10.924 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:29.917391+0000", 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00032674700000000001 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.119779+0000", 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202387337 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.925 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:805 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 30.375853415000002, 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51150757400000002, 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 805 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.926 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505063+0000", 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505067+0000", 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8139999999999998e-06 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.927 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505082+0000", 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4453e-05 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505124+0000", 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2849999999999998e-05 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505148+0000", 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3062999999999999e-05 2024-09-16T04:43:10.928 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611642+0000", 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106494218 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.611801+0000", 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000159338 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.929 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.613368+0000", 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001566462 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.711952+0000", 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098584630000000006 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016439+0000", 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30448657000000001 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.930 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.016571+0000", 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013217599999999999 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:806 3.10s0 3:0fc9379d:::smithi16530038-35:head [rollback 11] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 30.375720086000001, 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89166837600000004, 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.931 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 806 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.932 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505197+0000", 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505198+0000", 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7239999999999999e-06 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505210+0000", 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1688999999999999e-05 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.933 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505232+0000", 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1903999999999999e-05 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505249+0000", 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7036000000000001e-05 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017365+0000", 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.512115616 2024-09-16T04:43:10.934 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.017525+0000", 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016080200000000001 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.018648+0000", 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011226109999999999 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021794+0000", 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031461660000000002 2024-09-16T04:43:10.935 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396721+0000", 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.37492636899999998 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.396865+0000", 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014445899999999999 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.936 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:807 3.10s0 3:0fc9379d:::smithi16530038-35:head [append 0~0] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 30.375638763000001, 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.89196936699999996, 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 807 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.937 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505278+0000", 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505279+0000", 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3370000000000001e-06 2024-09-16T04:43:10.938 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505285+0000", 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0399999999999998e-06 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505308+0000", 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2595999999999998e-05 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505323+0000", 2024-09-16T04:43:10.939 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5342999999999999e-05 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019548+0000", 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51422475999999995 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.019822+0000", 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027388100000000002 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.940 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.021062+0000", 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001239614 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.122291+0000", 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101229427 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397057+0000", 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.27476627300000001 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.941 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.397247+0000", 2024-09-16T04:43:10.942 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000190096 2024-09-16T04:43:10.942 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.942 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.942 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.942 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.942 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:809 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~540672] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.363624351999999, 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70063091600000005, 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 809 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.943 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.944 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.944 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.944 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:43:10.944 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.944 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.944 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517292+0000", 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.517297+0000", 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8899999999999998e-06 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518377+0000", 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010801750000000001 2024-09-16T04:43:10.945 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518421+0000", 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3114000000000001e-05 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518443+0000", 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2782e-05 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518619+0000", 2024-09-16T04:43:10.946 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017519200000000001 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.518647+0000", 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8053999999999999e-05 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698939+0000", 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180292338 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.947 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.699242+0000", 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000302911 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.702023+0000", 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027812560000000002 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813606+0000", 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11158314900000001 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.948 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217721+0000", 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40411436000000001 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.217923+0000", 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000202695 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:810 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~720896] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:43:10.949 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.357619373999999, 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81676015000000002, 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 810 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.950 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523297+0000", 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.523300+0000", 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1140000000000002e-06 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.951 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524411+0000", 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001110307 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524443+0000", 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2067000000000003e-05 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524458+0000", 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5223e-05 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.952 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524644+0000", 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018582900000000001 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.524656+0000", 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1705999999999999e-05 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.953 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.704974+0000", 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.180318379 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.705144+0000", 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017024499999999999 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.706863+0000", 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00171842 2024-09-16T04:43:10.954 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813610+0000", 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10674731 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.339927+0000", 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52631677099999996 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340058+0000", 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130779 2024-09-16T04:43:10.955 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:811 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~786432] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.350966897999999, 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.81051717700000003, 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.956 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 811 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529950+0000", 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.957 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.529953+0000", 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.824e-06 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531687+0000", 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017341699999999999 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531721+0000", 2024-09-16T04:43:10.958 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3756e-05 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531744+0000", 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531894+0000", 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000150505 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:10.959 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531914+0000", 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0137e-05 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808146+0000", 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.276231536 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.808322+0000", 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000175899 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.960 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.810020+0000", 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016987230000000001 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813806+0000", 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0037855419999999998 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340306+0000", 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52649966299999995 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.961 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340467+0000", 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016146800000000001 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:812 3.22s0 3:456a3a59:::smithi16530038-43:head [append 0~49152] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.349134300999999, 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80902035800000005, 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.962 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 812 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.963 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531782+0000", 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531784+0000", 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.796e-06 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531896+0000", 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011207 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.964 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531918+0000", 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1412000000000001e-05 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531934+0000", 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6663999999999998e-05 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698290+0000", 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16635518999999999 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.965 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.698310+0000", 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0587000000000001e-05 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811428+0000", 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11311787800000001 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.811601+0000", 2024-09-16T04:43:10.966 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017340500000000001 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.812624+0000", 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001022659 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815470+0000", 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002845752 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.967 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340654+0000", 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52518383899999999 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.340803+0000", 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014910600000000001 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:813 3.22s0 3:456a3a59:::smithi16530038-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:43:10.968 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.34895581, 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80925618399999999, 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 813 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:43:10.969 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531961+0000", 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531962+0000", 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4339999999999999e-06 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.970 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531969+0000", 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7329999999999997e-06 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.531987+0000", 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7467000000000001e-05 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632132+0000", 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10014582499999999 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.813864+0000", 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18173177200000001 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814024+0000", 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016021500000000001 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.814762+0000", 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00073760799999999995 2024-09-16T04:43:10.972 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.816600+0000", 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001837679 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341069+0000", 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52446895500000001 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.973 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.341217+0000", 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000148496 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:814 3.22s0 3:456a3a59:::smithi16530038-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 29.248750968, 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.878598294, 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:10.974 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 814 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632166+0000", 2024-09-16T04:43:10.975 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632168+0000", 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632177+0000", 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9500000000000007e-06 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.976 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632196+0000", 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9296e-05 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.632207+0000", 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.137e-05 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815302+0000", 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.18309505200000001 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.977 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.815453+0000", 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000151065 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342052+0000", 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.52659867599999999 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.342192+0000", 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013975200000000001 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.978 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.979 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.979 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.510764+0000", 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.168572213 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1es0 e100,97 3:78f34faf:::smithi16530038-13:f,90'68)", 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:43:10.980 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 19.693461144, 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.63827547699999998, 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.981 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187456+0000", 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187458+0000", 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.12e-06 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187470+0000", 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2356e-05 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187502+0000", 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2135999999999999e-05 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.187519+0000", 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6730000000000001e-05 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.823533+0000", 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.63601388800000003 2024-09-16T04:43:10.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.825731+0000", 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002198247 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1es0 e100,97 3:78f34faf:::smithi16530038-13:11,92'72)", 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 19.582858019, 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.537458137, 2024-09-16T04:43:10.984 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298059+0000", 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298060+0000", 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.607e-06 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298071+0000", 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.03e-05 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298103+0000", 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2308999999999998e-05 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.986 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.298119+0000", 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5642999999999999e-05 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.834381+0000", 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.53626253700000004 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.835517+0000", 2024-09-16T04:43:10.987 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001135741 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:958 3.1es0 3:78f34faf:::smithi16530038-13:head [rollback 1d] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.285591195999999, 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54156909600000003, 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.988 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 958 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:43:10.989 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595326+0000", 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595327+0000", 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.257e-06 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.990 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595333+0000", 2024-09-16T04:43:10.991 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0959999999999997e-06 2024-09-16T04:43:10.991 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.991 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.991 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.991 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595353+0000", 2024-09-16T04:43:10.991 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9656999999999998e-05 2024-09-16T04:43:10.991 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595375+0000", 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1982999999999999e-05 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.722826+0000", 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12745103099999999 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.992 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.723141+0000", 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031585100000000002 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724986+0000", 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00184425 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.831289+0000", 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106303545 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.993 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.136810+0000", 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30552102599999997 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.136895+0000", 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4400000000000005e-05 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:959 3.1es0 3:78f34faf:::smithi16530038-13:head [append 0~0] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:43:10.994 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.285520296000001, 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.54178843700000001, 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 959 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595396+0000", 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595398+0000", 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.223e-06 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595404+0000", 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8239999999999996e-06 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595424+0000", 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0876000000000001e-05 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.595433+0000", 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9099999999999994e-06 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.827455+0000", 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.23202157200000001 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.827657+0000", 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000201709 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.828666+0000", 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010089389999999999 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.831586+0000", 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029207149999999999 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.137124+0000", 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30553806500000003 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.137185+0000", 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0603999999999998e-05 2024-09-16T04:43:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.000 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:11.001 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:43:11.002 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:43:11.002 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:43:11.003 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:43:11.003 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:43:11.003 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:43:11.004 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:43:11.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:43:11.004 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:43:11.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:43:11.005 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:43:11.005 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:10.994+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:11.016 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:43:11.016 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:43:11.041 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:43:11.041 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:43:11.065 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:43:11.065 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:43:11.087 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:43:11.087 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:43:11.102 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:11.098+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:11.111 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:43:11.112 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:43:11.135 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:43:11.135 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:43:11.135 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.15 2024-09-16T04:43:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:43:11.140 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:43:11.175 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:11.175 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:11.175 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:43:11.175 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:11.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:43:11.203 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:11.198+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:11.304 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:11.299+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:11.313 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:11.313 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:11.314 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:43:11.314 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:43:11.314 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:11.327 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:43:11.414 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:11.411+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.10s5 42/41 ECSubWrite(tid=29, reqid=client.4728.0:185, at_version=42'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 79.082637345999999, 2024-09-16T04:43:11.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20191694600000001, 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409854+0000", 2024-09-16T04:43:11.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409856+0000", 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0530000000000001e-06 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409917+0000", 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1023999999999997e-05 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.409958+0000", 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0945000000000002e-05 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510088+0000", 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100130017 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510247+0000", 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015866599999999999 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.510287+0000", 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0330999999999997e-05 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611689+0000", 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101401328 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:52.611771+0000", 2024-09-16T04:43:11.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2582000000000003e-05 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 60/32 ECSubWrite(tid=133, reqid=osd.4.0:133, at_version=60'37, trim_to=0'0, roll_forward_to=59'33))", 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 61.021087512999998, 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20246694400000001, 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.501 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471404+0000", 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471405+0000", 2024-09-16T04:43:11.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471443+0000", 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7605000000000003e-05 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.471463+0000", 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0452999999999999e-05 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.571565+0000", 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100102105 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671828+0000", 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10026227 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.671853+0000", 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5826e-05 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673802+0000", 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019481730000000001 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:10.673871+0000", 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9624999999999998e-05 2024-09-16T04:43:11.505 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 69/67 ECSubWrite(tid=149, reqid=client.4728.0:576, at_version=69'48, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 51.981546651000002, 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30202165600000003, 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510945+0000", 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510947+0000", 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.95e-06 2024-09-16T04:43:11.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.510983+0000", 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6198999999999997e-05 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.511014+0000", 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1112000000000003e-05 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.611120+0000", 2024-09-16T04:43:11.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100105729 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811521+0000", 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040100599999999 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.811553+0000", 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1958999999999997e-05 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812884+0000", 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013312700000000001 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.812967+0000", 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2430999999999997e-05 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.510 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 70/67 ECSubWrite(tid=155, reqid=osd.4.0:153, at_version=0'0, trim_to=0'0, roll_forward_to=70'52))", 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 50.926626151000001, 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137054300000001, 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.511 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565866+0000", 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565867+0000", 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.84e-06 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565893+0000", 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5715999999999999e-05 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.565933+0000", 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0358999999999998e-05 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666040+0000", 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010693499999999 2024-09-16T04:43:11.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666191+0000", 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015013299999999999 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.666251+0000", 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0615000000000002e-05 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767150+0000", 2024-09-16T04:43:11.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100898584 2024-09-16T04:43:11.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.515 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.515 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.516 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.767236+0000", 2024-09-16T04:43:11.516 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6360999999999995e-05 2024-09-16T04:43:11.516 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 74/69 ECSubWrite(tid=59, reqid=osd.8.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=73'49))", 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.989536332, 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26917078799999999, 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.517 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502955+0000", 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502956+0000", 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3e-07 2024-09-16T04:43:11.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.502980+0000", 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3802e-05 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.503017+0000", 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7082999999999997e-05 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603116+0000", 2024-09-16T04:43:11.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098623 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603238+0000", 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012259700000000001 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.603255+0000", 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6762999999999999e-05 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772084+0000", 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16882903799999999 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772126+0000", 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2151999999999999e-05 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=97, reqid=client.4728.0:613, at_version=74'68, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:43:11.521 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.925547512999998, 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20513274000000001, 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.522 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566944+0000", 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.566945+0000", 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.03e-07 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567199+0000", 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025406099999999998 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.523 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567228+0000", 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9589000000000001e-05 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567244+0000", 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6027000000000001e-05 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567424+0000", 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000180006 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567456+0000", 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1251999999999998e-05 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772018+0000", 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20456203000000001 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772077+0000", 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9372e-05 2024-09-16T04:43:11.525 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.es5 74/67 ECSubWrite(tid=98, reqid=client.4728.0:614, at_version=74'69, trim_to=0'0, roll_forward_to=72'66))", 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.925206994, 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.205026973, 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.526 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567285+0000", 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567286+0000", 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:43:11.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567509+0000", 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022361500000000001 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567536+0000", 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7063000000000002e-05 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.567555+0000", 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669312+0000", 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101756135 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.669353+0000", 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1069000000000003e-05 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772248+0000", 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102895556 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:24.772312+0000", 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3559000000000006e-05 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.530 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=74, reqid=client.4728.0:707, at_version=0'0, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.120217877999998, 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.202706724, 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372274+0000", 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372276+0000", 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.782e-06 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372297+0000", 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0907000000000002e-05 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372320+0000", 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3759000000000001e-05 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.372333+0000", 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2719e-05 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472559+0000", 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10022637399999999 2024-09-16T04:43:11.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.472570+0000", 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0351e-05 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574932+0000", 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10236213700000001 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574981+0000", 2024-09-16T04:43:11.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8695e-05 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 79/69 ECSubWrite(tid=75, reqid=client.4728.0:708, at_version=79'59, trim_to=0'0, roll_forward_to=79'58))", 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.117799476999998, 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20032509600000001, 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374692+0000", 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374694+0000", 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:43:11.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374738+0000", 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3016000000000003e-05 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374782+0000", 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4709999999999997e-05 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.374813+0000", 2024-09-16T04:43:11.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0700000000000001e-05 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473059+0000", 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098245669999999993 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.473071+0000", 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.24e-05 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.574989+0000", 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10191757 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:30.575017+0000", 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8773e-05 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=179, reqid=client.4728.0:716, at_version=81'60, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.952188358999997, 2024-09-16T04:43:11.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20327186, 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540303+0000", 2024-09-16T04:43:11.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540304+0000", 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.73e-07 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540825+0000", 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00052150999999999999 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540843+0000", 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8321e-05 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.540858+0000", 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.452e-05 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.741263+0000", 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20040504000000001 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.741281+0000", 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7946999999999999e-05 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.743496+0000", 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0022152199999999999 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.743575+0000", 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9128999999999994e-05 2024-09-16T04:43:11.543 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.bs4 81/67 ECSubWrite(tid=180, reqid=client.4728.0:717, at_version=81'61, trim_to=0'0, roll_forward_to=72'58))", 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.945807864999999, 2024-09-16T04:43:11.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29802481800000002, 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546684+0000", 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.546685+0000", 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4100000000000001e-06 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547131+0000", 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00044599499999999997 2024-09-16T04:43:11.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547146+0000", 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5145e-05 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.547155+0000", 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8710000000000003e-06 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.742419+0000", 2024-09-16T04:43:11.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.195263674 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.742437+0000", 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.84e-05 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.844629+0000", 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102192143 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:31.844709+0000", 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9179999999999997e-05 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es1 90/69 ECSubWrite(tid=84, reqid=osd.8.0:84, at_version=90'71, trim_to=0'0, roll_forward_to=82'67))", 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 30.900192135000001, 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19829587900000001, 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.549 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592300+0000", 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592301+0000", 2024-09-16T04:43:11.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5430000000000001e-06 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592337+0000", 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6075999999999998e-05 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592365+0000", 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7863000000000001e-05 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.592380+0000", 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.504e-05 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.788683+0000", 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19630244899999999 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.788698+0000", 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5113000000000001e-05 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.790504+0000", 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018063 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.790595+0000", 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1495000000000004e-05 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.8s6 92/55 ECSubWrite(tid=97, reqid=osd.5.0:97, at_version=92'31, trim_to=0'0, roll_forward_to=90'29))", 2024-09-16T04:43:11.553 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 28.888638520000001, 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.203223405, 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603853+0000", 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603854+0000", 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6700000000000003e-07 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603919+0000", 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5079000000000005e-05 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603953+0000", 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4215999999999999e-05 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.603970+0000", 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6626999999999999e-05 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.804328+0000", 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20035859 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.804344+0000", 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5252999999999999e-05 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.807006+0000", 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026628329999999999 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:42.807077+0000", 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0140000000000003e-05 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPush(3.1as5 101/97 [PushOp(3:5f4281cc:::smithi16530038-45:1a, version: 90'54, data_included: [262144~217088], data_size: 217088, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5f4281cc:::smithi16530038-45:1a@90'54, size: 1916928, copy_subset: [], clone_subset: {}, snapset: 1f=[]:{1a: [1a, 19, 18], 1f: [1f, 1e, 1d, 1b]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1916928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.592787+0000", 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 18.899704493000002, 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19797579700000001, 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.558 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.592787+0000", 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.592787+0000", 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.592788+0000", 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593214+0000", 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000425597 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593272+0000", 2024-09-16T04:43:11.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7960000000000001e-05 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593325+0000", 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3965999999999998e-05 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.789316+0000", 2024-09-16T04:43:11.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19599050200000001 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.790763+0000", 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001447039 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es5 104/97 ECSubWrite(tid=198, reqid=client.4728.0:957, at_version=104'94, trim_to=0'0, roll_forward_to=99'92))", 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.719871+0000", 2024-09-16T04:43:11.562 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.772621193999999, 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28552035100000001, 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719871+0000", 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719871+0000", 2024-09-16T04:43:11.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719872+0000", 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.813e-06 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719919+0000", 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6553e-05 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719953+0000", 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3912999999999999e-05 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.719969+0000", 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.66e-05 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.806281+0000", 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.08631192 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.565 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.806307+0000", 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5205999999999999e-05 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005331+0000", 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.199024957 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005391+0000", 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9388999999999997e-05 2024-09-16T04:43:11.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.1es5 104/97 ECSubWrite(tid=199, reqid=client.4728.0:958, at_version=104'95, trim_to=0'0, roll_forward_to=99'92))", 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.724906+0000", 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.767585389000001, 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28052565600000001, 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724906+0000", 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724906+0000", 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724908+0000", 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7069999999999999e-06 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724962+0000", 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3808999999999999e-05 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.724998+0000", 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5819999999999999e-05 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.725022+0000", 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4593e-05 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.808230+0000", 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.083207507 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.808257+0000", 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6812999999999999e-05 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005399+0000", 2024-09-16T04:43:11.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19714209399999999 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.005432+0000", 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3312999999999998e-05 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 105/67 ECSubWrite(tid=290, reqid=client.4728.0:975, at_version=105'111, trim_to=0'0, roll_forward_to=102'105))", 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.956133+0000", 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.536358610000001, 2024-09-16T04:43:11.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.201690813, 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956133+0000", 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956133+0000", 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956134+0000", 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.58e-07 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956157+0000", 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3388e-05 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956168+0000", 2024-09-16T04:43:11.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1374e-05 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956182+0000", 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3406e-05 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.056379+0000", 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100197366 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.056389+0000", 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.157733+0000", 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10134374 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.157824+0000", 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1089999999999997e-05 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.575 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 105/67 ECSubWrite(tid=292, reqid=osd.11.0:292, at_version=105'113, trim_to=0'0, roll_forward_to=102'105))", 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.956200+0000", 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.536291883000001, 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30249020900000001, 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956200+0000", 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956200+0000", 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956200+0000", 2024-09-16T04:43:11.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7300000000000001e-07 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956227+0000", 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7140000000000001e-05 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956239+0000", 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1426999999999999e-05 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956248+0000", 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0350000000000007e-06 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.156813+0000", 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.200565575 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.156847+0000", 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3309999999999998e-05 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.258574+0000", 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10172769600000001 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.258690+0000", 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000115553 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.cs4 105/67 ECSubWrite(tid=293, reqid=osd.11.0:293, at_version=105'115, trim_to=0'0, roll_forward_to=102'105))", 2024-09-16T04:43:11.580 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.956262+0000", 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.536230129, 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.303152486, 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956262+0000", 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.581 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956262+0000", 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956262+0000", 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3300000000000003e-07 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956288+0000", 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5797999999999999e-05 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.582 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956298+0000", 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0237e-05 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.956306+0000", 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9470000000000003e-06 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.257972+0000", 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30166649499999998 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.583 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.257990+0000", 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7496e-05 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.259345+0000", 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013545300000000001 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.259414+0000", 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9549999999999996e-05 2024-09-16T04:43:11.584 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.11s5 107/102 ECSubWrite(tid=58, reqid=osd.10.0:57, at_version=0'0, trim_to=0'0, roll_forward_to=107'60))", 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:57.931621+0000", 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.560871122, 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19922885800000001, 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:11.585 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:57.931621+0000", 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:57.931621+0000", 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:57.931622+0000", 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.26e-06 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.586 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:57.931645+0000", 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2768999999999998e-05 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:57.931669+0000", 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4672999999999999e-05 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:57.931690+0000", 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1039999999999998e-05 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:58.031894+0000", 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10020393499999999 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:58.031911+0000", 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6742999999999999e-05 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:58.130786+0000", 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098875008 2024-09-16T04:43:11.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:58.130849+0000", 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3429999999999994e-05 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:11.589 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:11.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:43:11.591 INFO:tasks.ceph.osd.13.smithi139.stderr:2024-09-16T04:43:11.590+0000 7f25bb099640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26302) UID: 0 2024-09-16T04:43:11.691 INFO:tasks.ceph.osd.4.smithi059.stderr:2024-09-16T04:43:11.686+0000 7ff1eee68640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26358) UID: 0 2024-09-16T04:43:11.720 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:43:11.764 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:11.764 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:11.764 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:43:11.764 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:11.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:43:11.792 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:43:11.790+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:43:11.893 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-09-16T04:43:11.891+0000 7f2e0510c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26661) UID: 0 2024-09-16T04:43:11.901 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:11.901 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:11.901 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:43:11.901 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:43:11.902 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:11.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:43:11.994 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:11.991+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=130, reqid=client.4728.0:530, at_version=65'45, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:43:12.041 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.784051421999997, 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.396784737, 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.042 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.252717+0000", 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.46e-07 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253567+0000", 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00085024700000000005 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.043 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253594+0000", 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6529e-05 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.253616+0000", 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2373000000000001e-05 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448095+0000", 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19447869800000001 2024-09-16T04:43:12.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.448103+0000", 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9379999999999992e-06 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649474+0000", 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20137053299999999 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.045 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649502+0000", 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7973000000000001e-05 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=131, reqid=client.4728.0:531, at_version=65'46, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.778353268000004, 2024-09-16T04:43:12.046 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.391121679, 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258415+0000", 2024-09-16T04:43:12.047 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258417+0000", 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.722e-06 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258658+0000", 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024151700000000001 2024-09-16T04:43:12.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258682+0000", 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3526000000000001e-05 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.258695+0000", 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3225e-05 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.049 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548388+0000", 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28969289100000001 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548393+0000", 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2970000000000003e-06 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649523+0000", 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101130161 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.050 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649537+0000", 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3339999999999999e-05 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=132, reqid=client.4728.0:532, at_version=65'47, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.775886628000002, 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38866693400000002, 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.051 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260882+0000", 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.052 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260883+0000", 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0750000000000001e-06 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260979+0000", 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6552999999999995e-05 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.260998+0000", 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9113000000000001e-05 2024-09-16T04:43:12.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.261010+0000", 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1817999999999999e-05 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548799+0000", 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28778851599999999 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.548839+0000", 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0371999999999999e-05 2024-09-16T04:43:12.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649539+0000", 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100699682 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649549+0000", 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8050000000000001e-06 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.055 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/32 ECSubWrite(tid=133, reqid=client.4728.0:533, at_version=65'48, trim_to=0'0, roll_forward_to=59'43))", 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.774248329999999, 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38775422500000001, 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:43:12.056 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262520+0000", 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262521+0000", 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.009e-06 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.057 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262542+0000", 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0981000000000001e-05 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262560+0000", 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8224e-05 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.262572+0000", 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2155e-05 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.058 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649287+0000", 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38671449499999999 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.649295+0000", 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8329999999999994e-06 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650254+0000", 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00095943799999999996 2024-09-16T04:43:12.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:15.650274+0000", 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0089999999999999e-05 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 70/53 ECSubWriteReply(tid=18, last_complete=70'32, committed=1, applied=1))", 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 51.613727963999999, 2024-09-16T04:43:12.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.32087775200000002, 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423040+0000", 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423042+0000", 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.77e-06 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423051+0000", 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8100000000000004e-06 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.423081+0000", 2024-09-16T04:43:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0626000000000001e-05 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.523175+0000", 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100093245 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.742603+0000", 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21942863300000001 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.743918+0000", 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001314668 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:937 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.580620+0000", 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.456147813000001, 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.132113506, 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 937 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580620+0000", 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580620+0000", 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580624+0000", 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4800000000000001e-06 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580657+0000", 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3176000000000002e-05 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580693+0000", 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6028999999999997e-05 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580711+0000", 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8377e-05 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580874+0000", 2024-09-16T04:43:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016261199999999999 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580891+0000", 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7263999999999999e-05 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.799454+0000", 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21856298099999999 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.799462+0000", 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6170000000000001e-06 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706333+0000", 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90687079400000004 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706338+0000", 2024-09-16T04:43:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5600000000000001e-06 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.912371+0000", 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.206032937 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for degraded object", 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.913944+0000", 2024-09-16T04:43:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015726609999999999 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.395897+0000", 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.481953138 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.396116+0000", 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00021892799999999999 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.397583+0000", 2024-09-16T04:43:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014675319999999999 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.502978+0000", 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105394321 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.712199+0000", 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20922146 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.712734+0000", 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00053463899999999995 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:938 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [rollback 1e] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.580748+0000", 2024-09-16T04:43:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.456020043999999, 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.232328887, 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 938 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580748+0000", 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580748+0000", 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580750+0000", 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.55e-06 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580757+0000", 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7300000000000005e-06 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580780+0000", 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2872999999999999e-05 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580796+0000", 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5722999999999998e-05 2024-09-16T04:43:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.581076+0000", 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027972400000000002 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.581101+0000", 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4749999999999999e-05 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800699+0000", 2024-09-16T04:43:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21959847099999999 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800704+0000", 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3639999999999998e-06 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706434+0000", 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90572917799999997 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.706439+0000", 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9840000000000001e-06 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914075+0000", 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2076361680000001 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for degraded object", 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914228+0000", 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000153549 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399055+0000", 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48482694199999998 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399427+0000", 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000372011 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.401491+0000", 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020635300000000001 2024-09-16T04:43:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.503165+0000", 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10167448699999999 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.812292+0000", 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30912664899999998 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813077+0000", 2024-09-16T04:43:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00078520399999999996 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:939 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 20.455950732000002, 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2334956030000002, 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 939 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580817+0000", 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580819+0000", 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.297e-06 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580825+0000", 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9499999999999998e-06 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.580864+0000", 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9606e-05 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.681043+0000", 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10017833499999999 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800804+0000", 2024-09-16T04:43:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11976181399999999 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:51.800808+0000", 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6430000000000001e-06 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.008117+0000", 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.207308388 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.008122+0000", 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5799999999999999e-06 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914306+0000", 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.906183721 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for degraded object", 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.914561+0000", 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025529200000000001 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.402786+0000", 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48822523899999998 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.403012+0000", 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000225448 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.404166+0000", 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011544509999999999 2024-09-16T04:43:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.503170+0000", 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099003718000000004 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813807+0000", 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31063750800000001 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.814313+0000", 2024-09-16T04:43:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00050561300000000003 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:943 3.5s0 3:a54da884:::smithi16530038-14:head [copy-from ver 0] snapc 22=[22,21,20,1f,1e,1d,1b,1a,19] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.584047+0000", 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 19.452721160999999, 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3330712170000001, 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 943 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584047+0000", 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584047+0000", 2024-09-16T04:43:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584051+0000", 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7359999999999999e-06 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584066+0000", 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.518e-05 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584103+0000", 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.748e-05 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584126+0000", 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2857000000000002e-05 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593972+0000", 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0098458580000000007 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.593979+0000", 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5769999999999999e-06 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.916665+0000", 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.322686647 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.919068+0000", 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024021889999999999 2024-09-16T04:43:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.026141+0000", 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10707359299999999 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.027046+0000", 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00090457999999999999 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.111730+0000", 2024-09-16T04:43:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.084683865999999997 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.112416+0000", 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00068637699999999995 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.917118+0000", 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80470227699999997 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:944 3.5s0 3:a54da884:::smithi16530038-14:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.584174+0000", 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 19.452594329, 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3352599700000001, 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 944 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584174+0000", 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584174+0000", 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584175+0000", 2024-09-16T04:43:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.587e-06 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584183+0000", 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2429999999999999e-06 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584206+0000", 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3637000000000001e-05 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.584224+0000", 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.738e-05 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.595503+0000", 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011279697999999999 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.595510+0000", 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1929999999999998e-06 2024-09-16T04:43:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.924019+0000", 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3285094630000001 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.924445+0000", 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00042564700000000003 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.027556+0000", 2024-09-16T04:43:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103110966 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.028442+0000", 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00088608700000000003 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.114030+0000", 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.085588417999999999 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for blocked object", 2024-09-16T04:43:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.114447+0000", 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00041656400000000002 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.917592+0000", 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.80314557499999994 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.918393+0000", 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000800707 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.919434+0000", 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010408049999999999 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=38, last_complete=38'14, committed=1, applied=1))", 2024-09-16T04:43:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.399580+0000", 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.637187835999999, 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31204127700000001, 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399580+0000", 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399580+0000", 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399582+0000", 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4869999999999999e-06 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399590+0000", 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2169999999999994e-06 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.399613+0000", 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2442999999999998e-05 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.499725+0000", 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100112408 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.711569+0000", 2024-09-16T04:43:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.211843632 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.711622+0000", 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3090000000000002e-05 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=40, last_complete=103'50, committed=1, applied=1))", 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.500039+0000", 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.536728969999999, 2024-09-16T04:43:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31329262800000002, 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500039+0000", 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500039+0000", 2024-09-16T04:43:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500040+0000", 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1699999999999997e-07 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500044+0000", 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9530000000000001e-06 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500055+0000", 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0672999999999999e-05 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.500062+0000", 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9439999999999999e-06 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813246+0000", 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.31318400800000001 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813332+0000", 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6333000000000002e-05 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=40, last_complete=38'14, committed=1, applied=1))", 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.504051+0000", 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.532717077000001, 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30950637600000003, 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504051+0000", 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504051+0000", 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504053+0000", 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.39e-06 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504060+0000", 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5100000000000001e-06 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504086+0000", 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6336999999999999e-05 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504101+0000", 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4382999999999999e-05 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813516+0000", 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30941474699999999 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813558+0000", 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2008999999999997e-05 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=39, last_complete=103'49, committed=1, applied=1))", 2024-09-16T04:43:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.504553+0000", 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.532215047000001, 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30856470699999999, 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504553+0000", 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504553+0000", 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504554+0000", 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7000000000000005e-07 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504558+0000", 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7759999999999999e-06 2024-09-16T04:43:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504571+0000", 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3773e-05 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.504580+0000", 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2640000000000008e-06 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.811967+0000", 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.307387615 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813118+0000", 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001150709 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 103/102 ECSubWriteReply(tid=40, last_complete=103'50, committed=1, applied=1))", 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:54.506035+0000", 2024-09-16T04:43:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 17.530733409, 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30937354099999997, 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506035+0000", 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506035+0000", 2024-09-16T04:43:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506036+0000", 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2270000000000001e-06 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506043+0000", 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3170000000000004e-06 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506066+0000", 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2195999999999999e-05 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.506078+0000", 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2807e-05 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.813628+0000", 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30755011399999999 2024-09-16T04:43:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:54.815408+0000", 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00177988 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:967 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.114620+0000", 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.922148654000001, 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41900611999999998, 2024-09-16T04:43:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 967 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114620+0000", 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114620+0000", 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114622+0000", 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.187e-06 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114631+0000", 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3530000000000001e-06 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114656+0000", 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.476e-05 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114676+0000", 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9706000000000001e-05 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.121725+0000", 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0070496040000000001 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.122126+0000", 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040128000000000001 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.124630+0000", 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002503365 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.129806+0000", 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051757340000000004 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.533049+0000", 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40324364499999998 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.533626+0000", 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00057648599999999997 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:968 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [rollback 23] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.114703+0000", 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.922064718000001, 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.420739427, 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 968 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114703+0000", 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114703+0000", 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114705+0000", 2024-09-16T04:43:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3069999999999999e-06 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114711+0000", 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4269999999999996e-06 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114732+0000", 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0780000000000001e-05 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114748+0000", 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5613000000000001e-05 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.126283+0000", 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011535665000000001 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.126717+0000", 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000433554 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.128150+0000", 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001433372 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.130216+0000", 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020659279999999999 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.534414+0000", 2024-09-16T04:43:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40419767499999998 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.535443+0000", 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010291059999999999 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:969 3.11s0 3:8c8bcb2a:::smithi16530038-46:head [append 0~0] snapc 24=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.114770+0000", 2024-09-16T04:43:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.921998098, 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.59494319200000001, 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 969 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114770+0000", 2024-09-16T04:43:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114770+0000", 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114771+0000", 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2300000000000001e-06 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114778+0000", 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1800000000000001e-06 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.114797+0000", 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9162999999999999e-05 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.214897+0000", 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009994799999999 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.218214+0000", 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0033174680000000001 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.218451+0000", 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023720499999999999 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.219546+0000", 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010942879999999999 2024-09-16T04:43:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.220601+0000", 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010550430000000001 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.709185+0000", 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.488584507 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.709713+0000", 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00052815999999999996 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGRecoveryDeleteReply(3.11s0 e105,102 3:882e132b:::smithi16530038-22:head,92'42)", 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.714221+0000", 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.322547052000001, 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39956194499999997, 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714221+0000", 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714221+0000", 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714222+0000", 2024-09-16T04:43:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.032e-06 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714230+0000", 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3560000000000004e-06 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714245+0000", 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5398999999999999e-05 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.714253+0000", 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5259999999999993e-06 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.112728+0000", 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.39847491299999999 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.113783+0000", 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001054719 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDECSubOpWrite(3.21s4 105/97 ECSubWrite(tid=229, reqid=osd.4.0:229, at_version=105'49, trim_to=0'0, roll_forward_to=99'47))", 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.304166072000001, 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.38255834700000002, 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732602+0000", 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.58e-07 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732631+0000", 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8413000000000001e-05 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732648+0000", 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6932999999999999e-05 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.732663+0000", 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4901e-05 2024-09-16T04:43:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.912643+0000", 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.17997981800000001 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:55.912656+0000", 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3485e-05 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.115112+0000", 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20245636 2024-09-16T04:43:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:56.115160+0000", 2024-09-16T04:43:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8078999999999998e-05 2024-09-16T04:43:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.152 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:12.153 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T04:43:12.154 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-09-16T04:43:12.146+0000 7f57048d9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26356) UID: 0 2024-09-16T04:43:12.255 INFO:tasks.ceph.osd.9.smithi139.stderr:2024-09-16T04:43:12.254+0000 7fa7739a9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26300) UID: 0 2024-09-16T04:43:12.328 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:12.328 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:12.328 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-09-16T04:43:12.328 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:12.343 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T04:43:12.390 INFO:tasks.ceph.osd.2.smithi144.stderr:2024-09-16T04:43:12.387+0000 7f7374c0e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26319) UID: 0 2024-09-16T04:43:12.390 INFO:tasks.ceph.osd.15.smithi165.stderr:2024-09-16T04:43:12.387+0000 7fb4c004e640 -1 osd.15 99 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:43:12.457 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:12.454+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:12.464 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:12.464 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-09-16T04:43:12.464 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-09-16T04:43:12.464 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-09-16T04:43:12.464 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:12.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T04:43:12.558 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:12.555+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:12.651 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-16T04:43:12.651 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-09-16T04:43:12.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-09-16T04:43:12.651 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-09-16T04:43:12.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.651 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:158 3.9s0 3:9de467d5:::smithi16530038-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 86.162222421999999, 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.51808159200000004, 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 158 2024-09-16T04:43:12.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484331+0000", 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484332+0000", 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1030000000000001e-06 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484339+0000", 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8630000000000002e-06 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484355+0000", 2024-09-16T04:43:12.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6473e-05 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484364+0000", 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8950000000000004e-06 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.586917+0000", 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102553035 2024-09-16T04:43:12.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.587177+0000", 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026005499999999998 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.588403+0000", 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001225152 2024-09-16T04:43:12.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.589911+0000", 2024-09-16T04:43:12.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015086260000000001 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002165+0000", 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41225355800000002 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.002413+0000", 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024783200000000002 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.9s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 86.162153613000001, 2024-09-16T04:43:12.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.62506688499999996, 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:43:12.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484400+0000", 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484401+0000", 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1200000000000001e-06 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484405+0000", 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.827e-06 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484416+0000", 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1609e-05 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.484425+0000", 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.790928+0000", 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30650327300000002 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:46.791067+0000", 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013860200000000001 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003346+0000", 2024-09-16T04:43:12.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.212279577 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.003611+0000", 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026456899999999998 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109440+0000", 2024-09-16T04:43:12.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105828857 2024-09-16T04:43:12.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.109467+0000", 2024-09-16T04:43:12.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6900999999999999e-05 2024-09-16T04:43:12.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.667 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.667 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:159 3.19s0 3:9de467d5:::smithi16530038-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 85.539201005999999, 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.976424197, 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.668 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 159 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107352+0000", 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107354+0000", 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9829999999999999e-06 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107362+0000", 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6129999999999998e-06 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.107388+0000", 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5675000000000001e-05 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207482+0000", 2024-09-16T04:43:12.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10009412299999999 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207644+0000", 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000161741 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:47.207649+0000", 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0359999999999997e-06 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082558+0000", 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8749096240000001 2024-09-16T04:43:12.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.082784+0000", 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00022614900000000001 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.083777+0000", 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099225300000000002 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:178 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:12.676 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 84.568439580000003, 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1069151189999999, 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.677 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.678 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 178 2024-09-16T04:43:12.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.678 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:43:12.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078114+0000", 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078116+0000", 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078127+0000", 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.062e-05 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078151+0000", 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4304999999999999e-05 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078175+0000", 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4006e-05 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776496+0000", 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.69832100500000005 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.776503+0000", 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.533e-06 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185014+0000", 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40851148700000001 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185029+0000", 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4780000000000001e-05 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:179 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 84.568344217999993, 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1068670979999999, 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.684 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 179 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078209+0000", 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078211+0000", 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.44e-06 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078220+0000", 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6299999999999993e-06 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078244+0000", 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3867999999999999e-05 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078260+0000", 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6203000000000001e-05 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780436+0000", 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70217526500000005 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.780441+0000", 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4620000000000004e-06 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185067+0000", 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40462590799999998 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185076+0000", 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3219999999999999e-06 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:180 3.19s0 3.b3ce8979 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 84.568271162000002, 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.106828843, 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.689 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 180 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078282+0000", 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078284+0000", 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8959999999999999e-06 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078291+0000", 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.657e-06 2024-09-16T04:43:12.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078310+0000", 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8885999999999998e-05 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.078325+0000", 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4772e-05 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981190+0000", 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90286589500000003 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.981194+0000", 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0790000000000002e-06 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185100+0000", 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20390586399999999 2024-09-16T04:43:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.185111+0000", 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0794e-05 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:185 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 84.547955182999999, 2024-09-16T04:43:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98955797199999995, 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:43:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098598+0000", 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098600+0000", 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1959999999999998e-06 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098611+0000", 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0423e-05 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098642+0000", 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1532000000000002e-05 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098658+0000", 2024-09-16T04:43:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5996000000000001e-05 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683726+0000", 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58506727400000003 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683731+0000", 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2730000000000002e-06 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088142+0000", 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40441113000000001 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.088156+0000", 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4148000000000001e-05 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:186 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 84.547867128000007, 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.089963279, 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 186 2024-09-16T04:43:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098686+0000", 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098687+0000", 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.063e-06 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098693+0000", 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6230000000000002e-06 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.098705+0000", 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2019e-05 2024-09-16T04:43:12.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198804+0000", 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100098513 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683799+0000", 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.48499579100000001 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.683801+0000", 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0889999999999998e-06 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188641+0000", 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.50483997300000005 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188650+0000", 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.208e-06 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.704 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:187 3.10s0 3.4a6e92d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 84.447717725000004, 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98988477799999997, 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 187 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198836+0000", 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198838+0000", 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.86e-06 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198846+0000", 2024-09-16T04:43:12.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0059999999999992e-06 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198864+0000", 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8468999999999999e-05 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.198876+0000", 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1443e-05 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783957+0000", 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.58508126900000001 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:48.783962+0000", 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7230000000000001e-06 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188710+0000", 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40474714899999997 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:41:49.188721+0000", 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0859000000000001e-05 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:583 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 53.312231769, 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2114807170000002, 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.710 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 583 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334322+0000", 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334325+0000", 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6969999999999999e-06 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334340+0000", 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4508999999999999e-05 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334388+0000", 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8016000000000002e-05 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.713 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334413+0000", 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5284999999999999e-05 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828480+0000", 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406639099999999 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828485+0000", 2024-09-16T04:43:12.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3430000000000003e-06 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.769267+0000", 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.94078222199999995 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.771858+0000", 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025913479999999998 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174292+0000", 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40243307299999997 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.174565+0000", 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027297299999999998 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.176259+0000", 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001694161 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281556+0000", 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10529764799999999 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.718 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.718 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545252+0000", 2024-09-16T04:43:12.718 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.263695658 2024-09-16T04:43:12.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.718 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.545802+0000", 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00055039300000000004 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:584 3.1as0 3:5f4281cc:::smithi16530038-45:head [rollback 11] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:43:12.719 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 53.312097305999998, 2024-09-16T04:43:12.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2123693160000002, 2024-09-16T04:43:12.720 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.720 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.720 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.720 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.720 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 584 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334456+0000", 2024-09-16T04:43:12.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334458+0000", 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.663e-06 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334465+0000", 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2239999999999998e-06 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334485+0000", 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9925999999999999e-05 2024-09-16T04:43:12.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334500+0000", 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5449000000000001e-05 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828570+0000", 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49407003700000002 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828576+0000", 2024-09-16T04:43:12.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.93e-06 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872096+0000", 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.043520019 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.872408+0000", 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031130200000000001 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177318+0000", 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30491066900000002 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.177579+0000", 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026102699999999999 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.178857+0000", 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001277395 2024-09-16T04:43:12.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281984+0000", 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10312769099999999 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546082+0000", 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26409753200000002 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.546825+0000", 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00074345200000000002 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:585 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~0] snapc 11=[11,f,d,b,a,7] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 53.312033434, 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3685253789999998, 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.728 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 585 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334520+0000", 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334521+0000", 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4169999999999999e-06 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334529+0000", 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8560000000000007e-06 2024-09-16T04:43:12.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334547+0000", 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7813e-05 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.334561+0000", 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3803e-05 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828622+0000", 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.49406153000000003 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:19.828624+0000", 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.277e-06 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873077+0000", 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.044453106 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for missing object", 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:20.873349+0000", 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027208400000000002 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279758+0000", 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40640956900000003 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.279965+0000", 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000206028 2024-09-16T04:43:12.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.281051+0000", 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010864399999999999 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.282237+0000", 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011862839999999999 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.701843+0000", 2024-09-16T04:43:12.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.41960623800000002 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:21.703045+0000", 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001201934 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:635 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~524288] snapc 15=[15,14,13,12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:43:12.735 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.473173825000003, 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.71243645600000005, 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 635 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:43:12.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173380+0000", 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.173382+0000", 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3829999999999999e-06 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.737 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174349+0000", 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00096718499999999996 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174408+0000", 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9036000000000002e-05 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.174432+0000", 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4176e-05 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579526+0000", 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40509398299999999 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.579719+0000", 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019240699999999999 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.581395+0000", 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016763769999999999 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.586083+0000", 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0046879950000000004 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885565+0000", 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29948223000000002 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:26.885816+0000", 2024-09-16T04:43:12.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025068399999999999 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:799 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~524288] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.153586775000001, 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.61375249600000004, 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.741 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 799 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492967+0000", 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.492970+0000", 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.027e-06 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.493969+0000", 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00099961899999999998 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494004+0000", 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5166000000000002e-05 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494056+0000", 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1807000000000001e-05 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494238+0000", 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018187800000000001 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.494252+0000", 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.349e-05 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.593780+0000", 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099528025000000006 2024-09-16T04:43:12.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.594015+0000", 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023556099999999999 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.596480+0000", 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024650140000000002 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.746 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803670+0000", 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.207189436 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.106478+0000", 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.30280800499999999 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.106719+0000", 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000241468 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.747 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:800 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~573440] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.149119194999997, 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70919317999999998, 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 800 2024-09-16T04:43:12.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497434+0000", 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.749 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.497437+0000", 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.796e-06 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498329+0000", 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000892124 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498363+0000", 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3908999999999997e-05 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.750 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498379+0000", 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6124e-05 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498519+0000", 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000140245 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.498535+0000", 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.579e-05 2024-09-16T04:43:12.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.599170+0000", 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100634526 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.599348+0000", 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017774199999999999 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.600830+0000", 2024-09-16T04:43:12.752 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001482178 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803675+0000", 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20284549599999999 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206492+0000", 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40281652400000001 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206627+0000", 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013572599999999999 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:801 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~688128] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.143917148, 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70428581800000001, 2024-09-16T04:43:12.754 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 801 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502636+0000", 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.502639+0000", 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2570000000000002e-06 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503645+0000", 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001006002 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503676+0000", 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1526999999999998e-05 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503692+0000", 2024-09-16T04:43:12.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5962999999999999e-05 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503829+0000", 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000137229 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.503845+0000", 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5270000000000001e-05 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.602063+0000", 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098218277000000007 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.602224+0000", 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000160996 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.603888+0000", 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016640330000000001 2024-09-16T04:43:12.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803679+0000", 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19979068899999999 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206810+0000", 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40313103500000003 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.206922+0000", 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011254 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:802 3.1as0 3:5f4281cc:::smithi16530038-45:head [append 0~294912] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.141786709999998, 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70333736700000005, 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.761 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 802 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504767+0000", 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.504768+0000", 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.48e-06 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505226+0000", 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00045742599999999998 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505255+0000", 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9810000000000001e-05 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505276+0000", 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0639999999999999e-05 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505379+0000", 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000103169 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505389+0000", 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8700000000000004e-06 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.798131+0000", 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.29274190900000002 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.798322+0000", 2024-09-16T04:43:12.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000190912 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.799632+0000", 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001310173 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.803863+0000", 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0042312249999999999 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.207875+0000", 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40401177799999999 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.208104+0000", 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000228975 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:803 3.1as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (57) in=7b] snapc 1a=[1a,19,18,11] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:43:12.767 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.141242298000002, 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.010178128, 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 803 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.768 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505311+0000", 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505313+0000", 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.533e-06 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505320+0000", 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6930000000000007e-06 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.505340+0000", 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.003e-05 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605448+0000", 2024-09-16T04:43:12.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10010718 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.210631+0000", 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.60518353999999996 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.210911+0000", 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00027993799999999998 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.212558+0000", 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016472839999999999 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.315431+0000", 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102872511 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.515321+0000", 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.19989022000000001 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.515489+0000", 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016819900000000001 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:804 3.1as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 32.041070742000002, 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3152593989999999, 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.773 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 804 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605483+0000", 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605485+0000", 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9230000000000001e-06 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605493+0000", 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4069999999999993e-06 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605516+0000", 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2506000000000001e-05 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:40.605528+0000", 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2734e-05 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.313900+0000", 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.70837135200000001 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.314072+0000", 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017190200000000001 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.516098+0000", 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20202645 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.516435+0000", 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000336535 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:41.920742+0000", 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.40430758999999999 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.778 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.4728.0:952 3.19s0 3:9e9173cd:::smithi16530038-4:head [create,setxattr kVnkrXsmOsSXXn[NoPlghhiQY]Z[N[YpkZhhkWli (1611) in=40b,setxattr mlQUZhlcqblpYXqOon[XkeRRf^]V]XgpQjbR]Onp (31) in=40b,setxattr mlhWpPgPW`YhakahTc_fjrboN^dbV\\geTbSTblQl (13680) in=40b,setxattr tnd`PUpci`\\YsrOok\\`[YnRfOdjtnTPaQ\\mmfZad (1754) in=40b,setxattr Ng^``Wsb`a]TmSYVrtjrbplbfeT (1787) in=27b,setxattr dVaqaN^[gldq]kbhcjaZasgTZ[Odhl]aQlfdOYTs (1154) in=40b,setxattr b\\bskO]Slnk\\Z[hXcNpRmeW^Oika`hnp[sjf^gZt (9120) in=40b,setxattr t]^h^V[jTXTXdlVSspZtqdq[YY_sZsYcTSUZ (1787) in=36b,setxattr aahNesb^`gaiRgYcaaOZVfi_pikNPgf\\kp^dVbgb (1997) in=40b,setxattr knq[rqmbh`VtiqaRolRdpfOZYiNUjt_U\\_NNfTiq (31) in=40b,setxattr mbZjYUYt`V_moWdtnSPfObZOZThTNm^fTSbtRmnk (1227) in=40b,setxattr aZashnRSif^YUbVsOs_YajObmqYNc`faT^mn\\ob] (1999) in=40b,setxattr ]VtQgeYWT_YgoePtsWnZO\\`d`b]\\]Oaqnmdlthe\\ (1997) in=40b,setxattr k_teaVkc]RthfgU\\jgWQ`SodpWYqY_nsj\\TqRecZ (66) in=40b,setxattr ropiQcfQSlPeYnYf^msbWfZkoY_m (1787) in=28b,setxattr X]VsnZ]]asTZVtojnVdigiarQe`^U^R^giahQ_la (210) in=40b,setxattr VaaZ\\mfqr^naQSibslaUjPm\\eptUP\\TOeNWaYWS[ (10104) in=40b,setxattr XNmgZfh`iqcZaNiTgXNNZskNflrnOePf__RVdanj (838) in=40b,setxattr nfk[Uja`tYW^hdPlPkRf[n[TeZ\\Oaes`qlh`Vfqp (1505) in=40b,setxattr oQO]kSdpW[PraahiWiph\\lsNqreWq]g]d_rsUVWk (15648) in=40b] snapc 23=[23,22,21,20,1f,1e,1d,1b,1a,19] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-09-16T04:42:52.658987+0000", 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 19.987566660999999, 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0521995120000001, 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "continuous": false, 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.4728", 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 952 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.779 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.658987+0000", 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.658987+0000", 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-09-16T04:43:12.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.658990+0000", 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0900000000000001e-06 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.659096+0000", 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010658900000000001 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.659133+0000", 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6502999999999999e-05 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:52.659156+0000", 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3430000000000001e-05 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.402976+0000", 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.743819644 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.403486+0000", 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00050984999999999995 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_started", 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.405869+0000", 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0023827919999999999 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_committed", 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.508664+0000", 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10279503700000001 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.710557+0000", 2024-09-16T04:43:12.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20189314799999999 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-09-16T04:42:53.711186+0000", 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00062942900000000003 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-16T04:43:12.784 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-16T04:43:12.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:43:12.786 INFO:tasks.ceph.osd.10.smithi144.stderr:2024-09-16T04:43:12.783+0000 7f64943e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26324) UID: 0 2024-09-16T04:43:12.887 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:12.886+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:12.917 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:43:12.917 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:43:12.917 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T04:43:12.917 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:43:12.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:43:12.975 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:12.987 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-16T04:43:12.983+0000 7f256eb3e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26658) UID: 0 2024-09-16T04:43:13.057 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:43:13.057 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T04:43:13.058 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T04:43:13.058 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T04:43:13.058 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:43:13.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:43:13.088 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-09-16T04:43:13.083+0000 7f6deffa6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26322) UID: 0 2024-09-16T04:43:13.141 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:43:13.190 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:13.186+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:13.240 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T04:43:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T04:43:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T04:43:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T04:43:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:122 3.as0 3:5619390f:::smithi16530038-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.558539007999997, 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.51059284400000005, 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678086+0000", 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678088+0000", 2024-09-16T04:43:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.12e-06 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678098+0000", 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0000000000000001e-05 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678119+0000", 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1676e-05 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678135+0000", 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5526999999999999e-05 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682759+0000", 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046242649999999998 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.682887+0000", 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012783599999999999 2024-09-16T04:43:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.683968+0000", 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010808090000000001 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.784795+0000", 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10082761899999999 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188432+0000", 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40363656399999998 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188678+0000", 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024642799999999998 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:123 3.as0 3:5619390f:::smithi16530038-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.558407672000001, 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91283640799999999, 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 123 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678217+0000", 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678219+0000", 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4030000000000001e-06 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678231+0000", 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678267+0000", 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5719000000000001e-05 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.678291+0000", 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4006e-05 2024-09-16T04:43:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684378+0000", 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060877320000000002 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.684496+0000", 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118076 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.188804+0000", 2024-09-16T04:43:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50430794400000001 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.189032+0000", 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000227235 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.591053+0000", 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.402021666 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:133 3.as0 3:5f4281cc:::smithi16530038-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.353219487000004, 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50755930599999999, 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883405+0000", 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.883408+0000", 2024-09-16T04:43:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8669999999999998e-06 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884235+0000", 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082694699999999997 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884270+0000", 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4690000000000002e-05 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884286+0000", 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6082999999999998e-05 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884441+0000", 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155852 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884838+0000", 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000396494 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.886865+0000", 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020266709999999999 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965465+0000", 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078600544999999994 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390790+0000", 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42532448499999997 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.390964+0000", 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017467200000000001 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:134 3.as0 3:5f4281cc:::smithi16530038-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.352307996999997, 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50704782699999995, 2024-09-16T04:43:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884317+0000", 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884318+0000", 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8299999999999995e-07 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884326+0000", 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.32e-06 2024-09-16T04:43:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884339+0000", 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.274e-05 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884348+0000", 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0229999999999998e-06 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888432+0000", 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040844899999999996 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.888564+0000", 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131419 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889204+0000", 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064011899999999997 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.965469+0000", 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076265854999999994 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391186+0000", 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.425716382 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.391364+0000", 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017849599999999999 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:135 3.as0 3:5f4281cc:::smithi16530038-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.352213900999999, 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0075776860000001, 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135 2024-09-16T04:43:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884411+0000", 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884412+0000", 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3399999999999997e-07 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884416+0000", 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3059999999999999e-06 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884428+0000", 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2500000000000001e-05 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.884439+0000", 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0662999999999999e-05 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889599+0000", 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051597179999999998 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:45.889722+0000", 2024-09-16T04:43:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012330300000000001 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392670+0000", 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50294774399999997 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.392905+0000", 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023530799999999999 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:46.891988+0000", 2024-09-16T04:43:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49908321 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:176 3.as0 3:5619390f:::smithi16530038-41:head [copy-from ver 10] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.161927122999998, 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71635610400000005, 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074697+0000", 2024-09-16T04:43:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074701+0000", 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9500000000000003e-06 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074716+0000", 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4523e-05 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074742+0000", 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5795999999999998e-05 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074764+0000", 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2646999999999998e-05 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074937+0000", 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017243599999999999 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074957+0000", 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9873000000000001e-05 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178498+0000", 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103540929 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178630+0000", 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132399 2024-09-16T04:43:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.493151+0000", 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.3145212 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.597069+0000", 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103917643 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.790921+0000", 2024-09-16T04:43:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19385244300000001 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.791054+0000", 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013226500000000001 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:177 3.as0 3:5619390f:::smithi16530038-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:43:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.161807589999995, 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42059460999999998, 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:43:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074817+0000", 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074819+0000", 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074828+0000", 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7399999999999993e-06 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074850+0000", 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2802999999999998e-05 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.074873+0000", 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2180000000000001e-05 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075093+0000", 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000220645 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.075109+0000", 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5823000000000001e-05 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178898+0000", 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10378908100000001 2024-09-16T04:43:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.178962+0000", 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489000000000001e-05 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495097+0000", 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31613566599999998 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495230+0000", 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133039 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495412+0000", 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018130800000000001 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:194 3.as0 3:5f4281cc:::smithi16530038-45:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:43:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.136648558000005, 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59800688199999996, 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 194 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:43:13.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099976+0000", 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099979+0000", 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6400000000000001e-06 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.099990+0000", 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.15e-05 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100022+0000", 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1569999999999998e-05 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.100039+0000", 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7486000000000002e-05 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179005+0000", 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078965628999999996 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.179189+0000", 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184155 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.486772+0000", 2024-09-16T04:43:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30758345500000001 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.495583+0000", 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088103490000000003 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697796+0000", 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20221325300000001 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:48.697983+0000", 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018684499999999999 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:276 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 80.137845073999998, 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71405619499999995, 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 276 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098779+0000", 2024-09-16T04:43:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098783+0000", 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098798+0000", 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5213e-05 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098845+0000", 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6853e-05 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098872+0000", 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6622999999999999e-05 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099054+0000", 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018249499999999999 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099075+0000", 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.065e-05 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207663+0000", 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858786400000001 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.207828+0000", 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016534199999999999 2024-09-16T04:43:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812836+0000", 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60500746100000002 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:277 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 80.137697222, 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81467523799999997, 2024-09-16T04:43:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 277 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098927+0000", 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098929+0000", 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4670000000000001e-06 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098938+0000", 2024-09-16T04:43:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098964+0000", 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5979999999999999e-05 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.098980+0000", 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5917e-05 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099168+0000", 2024-09-16T04:43:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000188228 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099176+0000", 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.307912+0000", 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20873598700000001 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.308068+0000", 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156448 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.913603+0000", 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60553407699999995 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:278 3.as0 3:5177b3bf:::smithi16530038-6:3 [read 0~98304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:43:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 80.137621112000005, 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81515018400000006, 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 278 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099003+0000", 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099005+0000", 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-06 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099012+0000", 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9e-06 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099031+0000", 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9303000000000001e-05 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099046+0000", 2024-09-16T04:43:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5003e-05 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099213+0000", 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000166773 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.099218+0000", 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9169999999999997e-06 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408464+0000", 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30924607700000001 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408612+0000", 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014793900000000001 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.914154+0000", 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50554194200000002 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpRead(3.as0 43/32 ECSubRead(tid=39, to_read={3:5177b3bf:::smithi16530038-6:head=0,24576,0}, subchunks={3:5177b3bf:::smithi16530038-6:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.827704283000003, 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40299000499999998, 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408920+0000", 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.408986+0000", 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409043+0000", 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6589999999999999e-05 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811767+0000", 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40272446200000001 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.811910+0000", 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014330300000000001 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=37, attrs_read=0))", 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.827304961999999, 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.403543547, 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409320+0000", 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409389+0000", 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9126000000000002e-05 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409407+0000", 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8097e-05 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812374+0000", 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40296689699999999 2024-09-16T04:43:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812863+0000", 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048942699999999996 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpReadReply(3.as0 43/32 ECSubReadReply(tid=39, attrs_read=0))", 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.826701727, 2024-09-16T04:43:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40309429400000002, 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409923+0000", 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409924+0000", 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1570000000000001e-06 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409963+0000", 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8753e-05 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409981+0000", 2024-09-16T04:43:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8737e-05 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.409991+0000", 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3889999999999994e-06 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.812922+0000", 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40293080100000001 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:41:53.813017+0000", 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5457000000000004e-05 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:704 3.14s0 3:28a14731:::smithi16530038-3:head [rollback d] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.575388535000002, 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40748865899999998, 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 704 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661236+0000", 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661237+0000", 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2100000000000001e-06 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661243+0000", 2024-09-16T04:43:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0970000000000002e-06 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661259+0000", 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5622999999999999e-05 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661270+0000", 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1182999999999999e-05 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.763754+0000", 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102483703 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.763924+0000", 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000170182 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.765124+0000", 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001200191 2024-09-16T04:43:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.767658+0000", 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025341579999999999 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068502+0000", 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30084363400000003 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068725+0000", 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000222678 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:705 3.14s0 3:28a14731:::smithi16530038-3:head [append 0~0] snapc 15=[12,11,d,b] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.575308327999998, 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40781490599999998, 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 705 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661316+0000", 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661317+0000", 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2729999999999999e-06 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661324+0000", 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3629999999999999e-06 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661335+0000", 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1396999999999999e-05 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.661346+0000", 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0546e-05 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.766069+0000", 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104723092 2024-09-16T04:43:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.766222+0000", 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015317900000000001 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.767028+0000", 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080575099999999995 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:29.868139+0000", 2024-09-16T04:43:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101110693 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.068984+0000", 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20084581600000001 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:30.069131+0000", 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014679600000000001 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:748 3.14s0 3:2b4d2b94:::smithi16530038-49:head [stat,delete] snapc 19=[19,18,17,16,11,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.791262089, 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.406000319, 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 748 2024-09-16T04:43:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445362+0000", 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445364+0000", 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445369+0000", 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4140000000000002e-06 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445388+0000", 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8292999999999999e-05 2024-09-16T04:43:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445402+0000", 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4620000000000001e-05 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445575+0000", 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017319199999999999 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T04:43:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.445594+0000", 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8363e-05 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.651371+0000", 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20577725199999999 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.651555+0000", 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018435899999999999 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.653237+0000", 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001681491 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.657570+0000", 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043331439999999997 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.851129+0000", 2024-09-16T04:43:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19355888900000001 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:35.851363+0000", 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023383200000000001 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDECSubOpWrite(3.12s5 97/53 ECSubWrite(tid=35, reqid=osd.9.0:35, at_version=97'36, trim_to=0'0, roll_forward_to=95'34))", 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.677545413000001, 2024-09-16T04:43:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.60339580100000001, 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:43:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559079+0000", 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.58e-07 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559112+0000", 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2073000000000003e-05 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559132+0000", 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0112999999999999e-05 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:47.559148+0000", 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5829e-05 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.060297+0000", 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50114971399999997 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.060328+0000", 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0593000000000003e-05 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.162396+0000", 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102068568 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:48.162475+0000", 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8553000000000005e-05 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.10.22:1 3.14s0 3:2b4d2b94:::smithi16530038-49:head [copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:54.115486+0000", 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.121138302999999, 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78126567199999997, 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115486+0000", 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115486+0000", 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115489+0000", 2024-09-16T04:43:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9799999999999998e-06 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115500+0000", 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0693000000000001e-05 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115522+0000", 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2419999999999999e-05 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115538+0000", 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5826999999999999e-05 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115722+0000", 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000183542 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.115728+0000", 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3260000000000001e-06 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.896273+0000", 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78054450099999995 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:54.896752+0000", 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000479383 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4728.0:980 3.14s0 3:2d731f27:::smithi16530038-30:head [create,setxattr qe_tlRrqOVZN]Sg`OogokNcdi`iTe[kSpgXo[tdi (347) in=40b,setxattr s_\\R^hs]`PhSfjan[pki[NgTNmQrWYik__[]RgPU (594) in=40b,setxattr `RVkQO\\NPp`^id``ktR\\nNYoo[aZVa[a (1401) in=32b,setxattr eUkd\\WaahPacqTYl^iV[kQsis\\aXTQqklkRsYTbs (873) in=40b,setxattr q\\qjarW`i[`bSljr`srl\\Vqni]Z`STgbdTnSmh`o (1963) in=40b,setxattr Sd^`oRTnVWgjdbTpljaQYrhO[``]a^haiQjgUoq` (18032) in=40b,setxattr nUtsh_fZqO^tgbj\\TdjN\\iNYcaa[UnfnknanjssP (1087) in=40b,setxattr _kcTXgOY]Up]^Vac[sbWiZ_aZifkbn`Vphdsslam (1434) in=40b,setxattr ttpkrgNZdj[]rR]iWqO]TaqdilgcW_tR\\^[_Pmgi (1083) in=40b,setxattr gYWdpfi^\\XWiW\\OjsQl_iOetb_TPV\\saa`i]Zkab (593) in=40b,setxattr qijT`_VpNN[TW`kts_Y_ngsTQrrOe_YNONO[^Qla (19224) in=40b,setxattr jXkpPeq`\\r\\hVgrkgqYiY_s[Up`[etXec^VWjblQ (175) in=40b,setxattr Ohgbf\\VQaSpfn^apdUcdOWhk]WUmXU\\`_NcacXQt (139) in=40b,setxattr [egW_laW[O\\dpqaiSbfc]Za^tico``qipqj`Pte` (1050) in=40b,setxattr h\\kdtVlaQPNaZakj[Y`XiYgiflaRcnt\\qnpaNURY (981) in=40b,setxattr as^bS\\_VZOPUeOaf\\]dtfgretlb]]`s``POt_SXc (19508) in=40b,setxattr NQg\\ajVioao__Weae_To_SsOrScUoVTbZRm[oc[O (1297) in=40b,setxattr UZTkha_tgerT`qq\\omSTjZSSXXZQU^f[RWr]`Znn (522) in=40b,setxattr dsdNaS_PtcaUjlnr]mXhVnPO_aZl]RWdh (1401) in=33b,setxattr pNnNhSVlYtVaX`q^i[ZgRflmt\\V_ZpabhitfmqXf (31) in=40b] snapc 24=[23,22,21,20,1f,1e,1d,1b,19] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T04:42:55.704172+0000", 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.532452761999998, 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.44758898600000002, 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4728", 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/334861116", 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 980 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704172+0000", 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704172+0000", 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704175+0000", 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8729999999999999e-06 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704223+0000", 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7966000000000001e-05 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704255+0000", 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2870000000000002e-05 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.704272+0000", 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6623e-05 2024-09-16T04:43:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.707855+0000", 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035828729999999999 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.708584+0000", 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072875499999999998 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T04:43:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.945732+0000", 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.237148472 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.945947+0000", 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000214489 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.947052+0000", 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011048060000000001 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:55.948684+0000", 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016326089999999999 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.151033+0000", 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20234907899999999 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T04:42:56.151761+0000", 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072757099999999999 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T04:43:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T04:43:13.347 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T04:43:13.348 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-09-16T04:43:13.348 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:43:13.350 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:13.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:43:13.351 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:43:13.351 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-09-16T04:43:13.352 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:43:13.352 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-16T04:43:13.028312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115991091,"num_objects":94,"num_object_clones":46,"num_object_copies":638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":7,"num_read":318,"num_read_kb":117971,"num_write":510,"num_write_kb":68822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":84115456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1912,"ondisk_log_size":1912,"up":256,"acting":256,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":751040,"kb_used_data":249908,"kb_used_omap":23,"kb_used_meta":500776,"kb_avail":1414826560,"statfs":{"total":1449551462400,"available":1448782397440,"internally_reserved":0,"allocated":255905792,"data_stored":304262791,"data_compressed":4249110,"data_compressed_allocated":5799936,"data_compressed_original":12460032,"omap_allocated":23928,"internal_metadata":512795272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001715"},"pg_stats":[{"pgid":"3.21","version":"105'49","reported_seq":111,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.832217+0000","last_change":"2024-09-16T04:42:57.726707+0000","last_active":"2024-09-16T04:43:08.832217+0000","last_peered":"2024-09-16T04:43:08.832217+0000","last_clean":"2024-09-16T04:43:08.832217+0000","last_became_active":"2024-09-16T04:42:49.166894+0000","last_became_peered":"2024-09-16T04:42:49.166894+0000","last_unstale":"2024-09-16T04:43:08.832217+0000","last_undegraded":"2024-09-16T04:43:08.832217+0000","last_fullsized":"2024-09-16T04:43:08.832217+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'31","last_scrub_stamp":"2024-09-16T04:42:26.688435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.688435+0000","objects_scrubbed":1,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:41.764873+0000","scrub_duration":1751,"objects_trimmed":0,"snaptrim_duration":0.103134376,"stat_sum":{"num_bytes":2375680,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.20","version":"36'4","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.833492+0000","last_change":"2024-09-16T04:43:00.834769+0000","last_active":"2024-09-16T04:43:08.833492+0000","last_peered":"2024-09-16T04:43:08.833492+0000","last_clean":"2024-09-16T04:43:08.833492+0000","last_became_active":"2024-09-16T04:42:53.909464+0000","last_became_peered":"2024-09-16T04:42:53.909464+0000","last_unstale":"2024-09-16T04:43:08.833492+0000","last_undegraded":"2024-09-16T04:43:08.833492+0000","last_fullsized":"2024-09-16T04:43:08.833492+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:43:00.834476+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:43:00.834476+0000","last_clean_scrub_stamp":"2024-09-16T04:43:00.834476+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:21.842968+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0.025490228,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11,1,5,6,4],"acting":[14,2,11,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1f","version":"105'64","reported_seq":105,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T04:43:07.703227+0000","last_change":"2024-09-16T04:42:57.828210+0000","last_active":"2024-09-16T04:43:07.703227+0000","last_peered":"2024-09-16T04:43:07.703227+0000","last_clean":"2024-09-16T04:43:07.703227+0000","last_became_active":"2024-09-16T04:42:48.879345+0000","last_became_peered":"2024-09-16T04:42:48.879345+0000","last_unstale":"2024-09-16T04:43:07.703227+0000","last_undegraded":"2024-09-16T04:43:07.703227+0000","last_fullsized":"2024-09-16T04:43:07.703227+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'52","last_scrub_stamp":"2024-09-16T04:42:25.895250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.895250+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:39.519034+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0.102254495,"stat_sum":{"num_bytes":1867776,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":10947,"num_write":5,"num_write_kb":1824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1867776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1e","version":"107'106","reported_seq":171,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.704524+0000","last_change":"2024-09-16T04:42:57.832704+0000","last_active":"2024-09-16T04:43:09.704524+0000","last_peered":"2024-09-16T04:43:09.704524+0000","last_clean":"2024-09-16T04:43:09.704524+0000","last_became_active":"2024-09-16T04:42:49.077358+0000","last_became_peered":"2024-09-16T04:42:49.077358+0000","last_unstale":"2024-09-16T04:43:09.704524+0000","last_undegraded":"2024-09-16T04:43:09.704524+0000","last_fullsized":"2024-09-16T04:43:09.704524+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"74'53","last_scrub_stamp":"2024-09-16T04:42:27.214170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.214170+0000","objects_scrubbed":5,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:51.357337+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0.109775179,"stat_sum":{"num_bytes":7618560,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":8932,"num_write":15,"num_write_kb":4464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6963200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1d","version":"108'169","reported_seq":202,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.696644+0000","last_change":"2024-09-16T04:43:00.154267+0000","last_active":"2024-09-16T04:43:08.696644+0000","last_peered":"2024-09-16T04:43:08.696644+0000","last_clean":"2024-09-16T04:43:08.696644+0000","last_became_active":"2024-09-16T04:42:58.938669+0000","last_became_peered":"2024-09-16T04:42:58.938669+0000","last_unstale":"2024-09-16T04:43:08.696644+0000","last_undegraded":"2024-09-16T04:43:08.696644+0000","last_fullsized":"2024-09-16T04:43:08.696644+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'133","last_scrub_stamp":"2024-09-16T04:42:36.357029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:36.357029+0000","objects_scrubbed":6,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:47.978805+0000","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0.30244215099999999,"stat_sum":{"num_bytes":11501568,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5332,"num_write":11,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":10862592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1c","version":"90'58","reported_seq":498,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T04:43:07.701768+0000","last_change":"2024-09-16T04:42:57.726957+0000","last_active":"2024-09-16T04:43:07.701768+0000","last_peered":"2024-09-16T04:43:07.701768+0000","last_clean":"2024-09-16T04:43:07.701768+0000","last_became_active":"2024-09-16T04:42:48.867904+0000","last_became_peered":"2024-09-16T04:42:48.867904+0000","last_unstale":"2024-09-16T04:43:07.701768+0000","last_undegraded":"2024-09-16T04:43:07.701768+0000","last_fullsized":"2024-09-16T04:43:07.701768+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":5,"last_scrub":"72'54","last_scrub_stamp":"2024-09-16T04:42:25.992723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.992723+0000","objects_scrubbed":2,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:35.687385+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001276606,"stat_sum":{"num_bytes":229376,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":673,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":229376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1b","version":"63'44","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.832537+0000","last_change":"2024-09-16T04:43:01.259748+0000","last_active":"2024-09-16T04:43:08.832537+0000","last_peered":"2024-09-16T04:43:08.832537+0000","last_clean":"2024-09-16T04:43:08.832537+0000","last_became_active":"2024-09-16T04:42:48.766598+0000","last_became_peered":"2024-09-16T04:42:48.766598+0000","last_unstale":"2024-09-16T04:43:08.832537+0000","last_undegraded":"2024-09-16T04:43:08.832537+0000","last_fullsized":"2024-09-16T04:43:08.832537+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'44","last_scrub_stamp":"2024-09-16T04:43:01.259286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.259286+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:22.843643+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0.003329668,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.1a","version":"105'69","reported_seq":130,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.702874+0000","last_change":"2024-09-16T04:42:57.827315+0000","last_active":"2024-09-16T04:43:09.702874+0000","last_peered":"2024-09-16T04:43:09.702874+0000","last_clean":"2024-09-16T04:43:09.702874+0000","last_became_active":"2024-09-16T04:42:49.062656+0000","last_became_peered":"2024-09-16T04:42:49.062656+0000","last_unstale":"2024-09-16T04:43:09.702874+0000","last_undegraded":"2024-09-16T04:43:09.702874+0000","last_fullsized":"2024-09-16T04:43:09.702874+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'51","last_scrub_stamp":"2024-09-16T04:42:35.060495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.060495+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:45.257678+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0.10248621199999999,"stat_sum":{"num_bytes":7405568,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":5360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5914624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.19","version":"102'46","reported_seq":484,"reported_epoch":115,"state":"active+undersized+degraded+remapped","last_fresh":"2024-09-16T04:43:08.733476+0000","last_change":"2024-09-16T04:42:54.104386+0000","last_active":"2024-09-16T04:43:08.733476+0000","last_peered":"2024-09-16T04:43:08.733476+0000","last_clean":"2024-09-16T04:42:06.828625+0000","last_became_active":"2024-09-16T04:42:53.401433+0000","last_became_peered":"2024-09-16T04:42:53.401433+0000","last_unstale":"2024-09-16T04:43:08.733476+0000","last_undegraded":"2024-09-16T04:42:53.508182+0000","last_fullsized":"2024-09-16T04:42:52.588994+0000","mapping_epoch":101,"log_start":"44'22","ondisk_log_start":"44'22","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'26","last_scrub_stamp":"2024-09-16T04:42:00.003872+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.003872+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":22,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.098972921000000005,"stat_sum":{"num_bytes":6340608,"num_objects":10,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":11622,"num_write":29,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":9273344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11,9,13,7,1],"acting":[14,1,0,3,2147483647,8,10],"avail_no_missing":["14(0)","0(2)","1(1)","3(3)","8(5)","10(6)"],"object_location_counts":[{"shards":"0(2),1(1),3(3),8(5),10(6),14(0)","objects":10}],"blocked_by":[],"up_primary":8,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.18","version":"44'8","reported_seq":444,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.710095+0000","last_change":"2024-09-16T04:42:57.826940+0000","last_active":"2024-09-16T04:43:10.710095+0000","last_peered":"2024-09-16T04:43:10.710095+0000","last_clean":"2024-09-16T04:43:10.710095+0000","last_became_active":"2024-09-16T04:42:12.629523+0000","last_became_peered":"2024-09-16T04:42:12.629523+0000","last_unstale":"2024-09-16T04:43:10.710095+0000","last_undegraded":"2024-09-16T04:43:10.710095+0000","last_fullsized":"2024-09-16T04:43:10.710095+0000","mapping_epoch":61,"log_start":"44'8","ondisk_log_start":"44'8","created":55,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'8","last_scrub_stamp":"2024-09-16T04:42:14.378029+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:14.378029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:41.738936+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0.101625202,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,0,4,5,7,12],"acting":[13,6,0,4,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.17","version":"105'12","reported_seq":324,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.703165+0000","last_change":"2024-09-16T04:42:57.623204+0000","last_active":"2024-09-16T04:43:09.703165+0000","last_peered":"2024-09-16T04:43:09.703165+0000","last_clean":"2024-09-16T04:43:09.703165+0000","last_became_active":"2024-09-16T04:42:04.604172+0000","last_became_peered":"2024-09-16T04:42:04.604172+0000","last_unstale":"2024-09-16T04:43:09.703165+0000","last_undegraded":"2024-09-16T04:43:09.703165+0000","last_fullsized":"2024-09-16T04:43:09.703165+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'6","last_scrub_stamp":"2024-09-16T04:42:11.540212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:11.540212+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:11.728434+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0.0023141920000000001,"stat_sum":{"num_bytes":606208,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":3554,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,10,9,7,1,13],"acting":[5,3,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.16","version":"92'16","reported_seq":317,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T04:43:07.702749+0000","last_change":"2024-09-16T04:42:57.827452+0000","last_active":"2024-09-16T04:43:07.702749+0000","last_peered":"2024-09-16T04:43:07.702749+0000","last_clean":"2024-09-16T04:43:07.702749+0000","last_became_active":"2024-09-16T04:42:53.801354+0000","last_became_peered":"2024-09-16T04:42:53.801354+0000","last_unstale":"2024-09-16T04:43:07.702749+0000","last_undegraded":"2024-09-16T04:43:07.702749+0000","last_fullsized":"2024-09-16T04:43:07.702749+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:57.977558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:57.977558+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:17.776743+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.101716587,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":81920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4,8,9,5,10],"acting":[11,1,4,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.15","version":"36'6","reported_seq":330,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.806719+0000","last_change":"2024-09-16T04:42:57.720101+0000","last_active":"2024-09-16T04:43:10.806719+0000","last_peered":"2024-09-16T04:43:10.806719+0000","last_clean":"2024-09-16T04:43:10.806719+0000","last_became_active":"2024-09-16T04:42:54.005049+0000","last_became_peered":"2024-09-16T04:42:54.005049+0000","last_unstale":"2024-09-16T04:43:10.806719+0000","last_undegraded":"2024-09-16T04:43:10.806719+0000","last_fullsized":"2024-09-16T04:43:10.806719+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'6","last_scrub_stamp":"2024-09-16T04:42:10.772708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.772708+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:15.673676+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.0022826220000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3,13,14,2,1],"acting":[9,6,3,13,14,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.14","version":"111'105","reported_seq":453,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.703013+0000","last_change":"2024-09-16T04:43:01.656887+0000","last_active":"2024-09-16T04:43:09.703013+0000","last_peered":"2024-09-16T04:43:09.703013+0000","last_clean":"2024-09-16T04:43:09.703013+0000","last_became_active":"2024-09-16T04:43:01.352468+0000","last_became_peered":"2024-09-16T04:43:01.352468+0000","last_unstale":"2024-09-16T04:43:09.703013+0000","last_undegraded":"2024-09-16T04:43:09.703013+0000","last_fullsized":"2024-09-16T04:43:09.703013+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'25","last_scrub_stamp":"2024-09-16T04:41:58.555869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.555869+0000","objects_scrubbed":5,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":899,"objects_trimmed":0,"snaptrim_duration":0.202399415,"stat_sum":{"num_bytes":8716288,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":9,"num_read_kb":5332,"num_write":41,"num_write_kb":7280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":14368768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,8,1,13,5,4],"acting":[3,9,8,1,13,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.710895+0000","last_change":"2024-09-16T04:42:53.671490+0000","last_active":"2024-09-16T04:43:10.710895+0000","last_peered":"2024-09-16T04:43:10.710895+0000","last_clean":"2024-09-16T04:43:10.710895+0000","last_became_active":"2024-09-16T04:42:53.671176+0000","last_became_peered":"2024-09-16T04:42:53.671176+0000","last_unstale":"2024-09-16T04:43:10.710895+0000","last_undegraded":"2024-09-16T04:43:10.710895+0000","last_fullsized":"2024-09-16T04:43:10.710895+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:00.461060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8],"acting":[13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"35'2","reported_seq":380,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T04:43:07.743383+0000","last_change":"2024-09-16T04:42:57.617205+0000","last_active":"2024-09-16T04:43:07.743383+0000","last_peered":"2024-09-16T04:43:07.743383+0000","last_clean":"2024-09-16T04:43:07.743383+0000","last_became_active":"2024-09-16T04:42:53.798280+0000","last_became_peered":"2024-09-16T04:42:53.798280+0000","last_unstale":"2024-09-16T04:43:07.743383+0000","last_undegraded":"2024-09-16T04:43:07.743383+0000","last_fullsized":"2024-09-16T04:43:07.743383+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:41:50.605455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:50.605455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:11.135823+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0.0013274440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,14,12,7,9],"acting":[0,13,1,14,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":193,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.795038+0000","last_change":"2024-09-16T04:41:30.072090+0000","last_active":"2024-09-16T04:43:08.795038+0000","last_peered":"2024-09-16T04:43:08.795038+0000","last_clean":"2024-09-16T04:43:08.795038+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:43:08.795038+0000","last_undegraded":"2024-09-16T04:43:08.795038+0000","last_fullsized":"2024-09-16T04:43:08.795038+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:13.659676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'2","reported_seq":379,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.832930+0000","last_change":"2024-09-16T04:42:57.627200+0000","last_active":"2024-09-16T04:43:08.832930+0000","last_peered":"2024-09-16T04:43:08.832930+0000","last_clean":"2024-09-16T04:43:08.832930+0000","last_became_active":"2024-09-16T04:41:48.704091+0000","last_became_peered":"2024-09-16T04:41:48.704091+0000","last_unstale":"2024-09-16T04:43:08.832930+0000","last_undegraded":"2024-09-16T04:43:08.832930+0000","last_fullsized":"2024-09-16T04:43:08.832930+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-16T04:42:00.916431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:00.916431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:26.848432+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.0023689520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.709472+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:43:09.709472+0000","last_peered":"2024-09-16T04:43:09.709472+0000","last_clean":"2024-09-16T04:43:09.709472+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:43:09.709472+0000","last_undegraded":"2024-09-16T04:43:09.709472+0000","last_fullsized":"2024-09-16T04:43:09.709472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"107'31","reported_seq":419,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.695044+0000","last_change":"2024-09-16T04:42:58.139380+0000","last_active":"2024-09-16T04:43:08.695044+0000","last_peered":"2024-09-16T04:43:08.695044+0000","last_clean":"2024-09-16T04:43:08.695044+0000","last_became_active":"2024-09-16T04:41:48.775737+0000","last_became_peered":"2024-09-16T04:41:48.775737+0000","last_unstale":"2024-09-16T04:43:08.695044+0000","last_undegraded":"2024-09-16T04:43:08.695044+0000","last_fullsized":"2024-09-16T04:43:08.695044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'10","last_scrub_stamp":"2024-09-16T04:41:59.997918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:59.997918+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:26.803273+0000","scrub_duration":1143,"objects_trimmed":0,"snaptrim_duration":0.30758759000000002,"stat_sum":{"num_bytes":5210112,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":11,"num_write_kb":4496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":199,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.932815+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:43:08.932815+0000","last_peered":"2024-09-16T04:43:08.932815+0000","last_clean":"2024-09-16T04:43:08.932815+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:43:08.932815+0000","last_undegraded":"2024-09-16T04:43:08.932815+0000","last_fullsized":"2024-09-16T04:43:08.932815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"97'84","reported_seq":83,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.070600+0000","last_change":"2024-09-16T04:42:57.928525+0000","last_active":"2024-09-16T04:43:10.070600+0000","last_peered":"2024-09-16T04:43:10.070600+0000","last_clean":"2024-09-16T04:43:10.070600+0000","last_became_active":"2024-09-16T04:42:53.912853+0000","last_became_peered":"2024-09-16T04:42:53.912853+0000","last_unstale":"2024-09-16T04:43:10.070600+0000","last_undegraded":"2024-09-16T04:43:10.070600+0000","last_fullsized":"2024-09-16T04:43:10.070600+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-16T04:41:58.060970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:41:58.060970+0000","objects_scrubbed":4,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:22.162333+0000","scrub_duration":691,"objects_trimmed":0,"snaptrim_duration":0.102446947,"stat_sum":{"num_bytes":4980736,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":676,"num_write":36,"num_write_kb":4208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,13,7,0,6],"acting":[10,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.3","version":"97'43","reported_seq":477,"reported_epoch":117,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T04:43:11.022096+0000","last_change":"2024-09-16T04:43:10.542014+0000","last_active":"2024-09-16T04:43:11.022096+0000","last_peered":"2024-09-16T04:43:11.022096+0000","last_clean":"2024-09-16T04:43:11.022096+0000","last_became_active":"2024-09-16T04:43:01.458966+0000","last_became_peered":"2024-09-16T04:43:01.458966+0000","last_unstale":"2024-09-16T04:43:11.022096+0000","last_undegraded":"2024-09-16T04:43:11.022096+0000","last_fullsized":"2024-09-16T04:43:11.022096+0000","mapping_epoch":102,"log_start":"97'43","ondisk_log_start":"97'43","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'21","last_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_deep_scrub":"72'21","last_deep_scrub_stamp":"2024-09-16T04:42:27.317975+0000","last_clean_scrub_stamp":"2024-09-16T04:42:27.317975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":48,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0.309208755,"stat_sum":{"num_bytes":5120000,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":22564,"num_write":8,"num_write_kb":4592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5668864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,9,2,5,3],"acting":[13,7,4,9,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.2","version":"28'2","reported_seq":205,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.703296+0000","last_change":"2024-09-16T04:42:46.312976+0000","last_active":"2024-09-16T04:43:09.703296+0000","last_peered":"2024-09-16T04:43:09.703296+0000","last_clean":"2024-09-16T04:43:09.703296+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:43:09.703296+0000","last_undegraded":"2024-09-16T04:43:09.703296+0000","last_fullsized":"2024-09-16T04:43:09.703296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:42:46.312734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:46.312734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:49.120009+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"36'4","reported_seq":404,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.833808+0000","last_change":"2024-09-16T04:42:57.640829+0000","last_active":"2024-09-16T04:43:08.833808+0000","last_peered":"2024-09-16T04:43:08.833808+0000","last_clean":"2024-09-16T04:43:08.833808+0000","last_became_active":"2024-09-16T04:42:53.908802+0000","last_became_peered":"2024-09-16T04:42:53.908802+0000","last_unstale":"2024-09-16T04:43:08.833808+0000","last_undegraded":"2024-09-16T04:43:08.833808+0000","last_fullsized":"2024-09-16T04:43:08.833808+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'4","last_scrub_stamp":"2024-09-16T04:42:26.089771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.089771+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:47.946800+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.001566668,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11,1,5,6,4],"acting":[14,2,11,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":196,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.810762+0000","last_change":"2024-09-16T04:41:30.209135+0000","last_active":"2024-09-16T04:43:10.810762+0000","last_peered":"2024-09-16T04:43:10.810762+0000","last_clean":"2024-09-16T04:43:10.810762+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:43:10.810762+0000","last_undegraded":"2024-09-16T04:43:10.810762+0000","last_fullsized":"2024-09-16T04:43:10.810762+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"105'59","reported_seq":464,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.932701+0000","last_change":"2024-09-16T04:42:57.725943+0000","last_active":"2024-09-16T04:43:08.932701+0000","last_peered":"2024-09-16T04:43:08.932701+0000","last_clean":"2024-09-16T04:43:08.932701+0000","last_became_active":"2024-09-16T04:42:18.991363+0000","last_became_peered":"2024-09-16T04:42:18.991363+0000","last_unstale":"2024-09-16T04:43:08.932701+0000","last_undegraded":"2024-09-16T04:43:08.932701+0000","last_fullsized":"2024-09-16T04:43:08.932701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'35","last_scrub_stamp":"2024-09-16T04:42:24.144068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.144068+0000","objects_scrubbed":2,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:48.776778+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0.0014621390000000001,"stat_sum":{"num_bytes":5275648,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":195,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.702747+0000","last_change":"2024-09-16T04:41:29.618967+0000","last_active":"2024-09-16T04:43:09.702747+0000","last_peered":"2024-09-16T04:43:09.702747+0000","last_clean":"2024-09-16T04:43:09.702747+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:43:09.702747+0000","last_undegraded":"2024-09-16T04:43:09.702747+0000","last_fullsized":"2024-09-16T04:43:09.702747+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:41:26.370698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:13.141719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"111'35","reported_seq":434,"reported_epoch":117,"state":"active","last_fresh":"2024-09-16T04:43:11.052199+0000","last_change":"2024-09-16T04:43:11.052199+0000","last_active":"2024-09-16T04:43:11.052199+0000","last_peered":"2024-09-16T04:43:11.052199+0000","last_clean":"2024-09-16T04:43:09.309424+0000","last_became_active":"2024-09-16T04:43:11.052199+0000","last_became_peered":"2024-09-16T04:43:11.052199+0000","last_unstale":"2024-09-16T04:43:11.052199+0000","last_undegraded":"2024-09-16T04:43:11.052199+0000","last_fullsized":"2024-09-16T04:43:11.052199+0000","mapping_epoch":102,"log_start":"111'35","ondisk_log_start":"111'35","created":32,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'15","last_scrub_stamp":"2024-09-16T04:42:26.595325+0000","last_deep_scrub":"36'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.564481+0000","last_clean_scrub_stamp":"2024-09-16T04:42:26.595325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":33,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":967,"objects_trimmed":0,"snaptrim_duration":0.104938434,"stat_sum":{"num_bytes":6373376,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12566528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,12,1,11],"acting":[13,2,3,6,12,1,11],"avail_no_missing":["13(0)","1(5)","2(1)","3(2)","6(3)","11(6)","12(4)"],"object_location_counts":[{"shards":"1(5),2(1),3(2),6(3),11(6),12(4),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"2.3","version":"111'2","reported_seq":213,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.702910+0000","last_change":"2024-09-16T04:43:01.980148+0000","last_active":"2024-09-16T04:43:09.702910+0000","last_peered":"2024-09-16T04:43:09.702910+0000","last_clean":"2024-09-16T04:43:09.702910+0000","last_became_active":"2024-09-16T04:42:54.097360+0000","last_became_peered":"2024-09-16T04:42:54.097360+0000","last_unstale":"2024-09-16T04:43:09.702910+0000","last_undegraded":"2024-09-16T04:43:09.702910+0000","last_fullsized":"2024-09-16T04:43:09.702910+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.979750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:31.635264+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,13],"acting":[5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":399,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.710498+0000","last_change":"2024-09-16T04:42:36.062121+0000","last_active":"2024-09-16T04:43:10.710498+0000","last_peered":"2024-09-16T04:43:10.710498+0000","last_clean":"2024-09-16T04:43:10.710498+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:43:10.710498+0000","last_undegraded":"2024-09-16T04:43:10.710498+0000","last_fullsized":"2024-09-16T04:43:10.710498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:42:36.061866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:42:36.061866+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:37.369149+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"107'35","reported_seq":468,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.703122+0000","last_change":"2024-09-16T04:42:58.357253+0000","last_active":"2024-09-16T04:43:09.703122+0000","last_peered":"2024-09-16T04:43:09.703122+0000","last_clean":"2024-09-16T04:43:09.703122+0000","last_became_active":"2024-09-16T04:42:57.944513+0000","last_became_peered":"2024-09-16T04:42:57.944513+0000","last_unstale":"2024-09-16T04:43:09.703122+0000","last_undegraded":"2024-09-16T04:43:09.703122+0000","last_fullsized":"2024-09-16T04:43:09.703122+0000","mapping_epoch":106,"log_start":"44'8","ondisk_log_start":"44'8","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-09-16T04:42:13.576020+0000","last_deep_scrub":"38'4","last_deep_scrub_stamp":"2024-09-16T04:41:50.115829+0000","last_clean_scrub_stamp":"2024-09-16T04:42:13.576020+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":12,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:42.557108+0000","scrub_duration":1403,"objects_trimmed":0,"snaptrim_duration":0.310307373,"stat_sum":{"num_bytes":1982464,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1982464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,10,6],"acting":[5,0,2,1,14,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.9","version":"50'30","reported_seq":453,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.733511+0000","last_change":"2024-09-16T04:43:00.464849+0000","last_active":"2024-09-16T04:43:08.733511+0000","last_peered":"2024-09-16T04:43:08.733511+0000","last_clean":"2024-09-16T04:43:08.733511+0000","last_became_active":"2024-09-16T04:43:00.361000+0000","last_became_peered":"2024-09-16T04:43:00.361000+0000","last_unstale":"2024-09-16T04:43:08.733511+0000","last_undegraded":"2024-09-16T04:43:08.733511+0000","last_fullsized":"2024-09-16T04:43:08.733511+0000","mapping_epoch":109,"log_start":"44'22","ondisk_log_start":"44'22","created":32,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'30","last_scrub_stamp":"2024-09-16T04:42:10.554025+0000","last_deep_scrub":"36'11","last_deep_scrub_stamp":"2024-09-16T04:41:51.621244+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.554025+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":22,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:25.187693+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.101595397,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,4,8,10],"acting":[14,1,0,3,4,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.a","version":"97'77","reported_seq":481,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.703218+0000","last_change":"2024-09-16T04:42:57.825659+0000","last_active":"2024-09-16T04:43:09.703218+0000","last_peered":"2024-09-16T04:43:09.703218+0000","last_clean":"2024-09-16T04:43:09.703218+0000","last_became_active":"2024-09-16T04:42:18.694273+0000","last_became_peered":"2024-09-16T04:42:18.694273+0000","last_unstale":"2024-09-16T04:43:09.703218+0000","last_undegraded":"2024-09-16T04:43:09.703218+0000","last_fullsized":"2024-09-16T04:43:09.703218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'41","last_scrub_stamp":"2024-09-16T04:42:25.530017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.530017+0000","objects_scrubbed":5,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:31.594139+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.100560026,"stat_sum":{"num_bytes":360448,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":1372,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.b","version":"105'83","reported_seq":474,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.932936+0000","last_change":"2024-09-16T04:42:57.630747+0000","last_active":"2024-09-16T04:43:08.932936+0000","last_peered":"2024-09-16T04:43:08.932936+0000","last_clean":"2024-09-16T04:43:08.932936+0000","last_became_active":"2024-09-16T04:42:18.688040+0000","last_became_peered":"2024-09-16T04:42:18.688040+0000","last_unstale":"2024-09-16T04:43:08.932936+0000","last_undegraded":"2024-09-16T04:43:08.932936+0000","last_fullsized":"2024-09-16T04:43:08.932936+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'58","last_scrub_stamp":"2024-09-16T04:42:25.280206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.280206+0000","objects_scrubbed":6,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:31.941002+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0.0035392930000000002,"stat_sum":{"num_bytes":4964352,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":9112,"num_write":6,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.c","version":"107'119","reported_seq":498,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T04:43:07.702109+0000","last_change":"2024-09-16T04:42:58.240919+0000","last_active":"2024-09-16T04:43:07.702109+0000","last_peered":"2024-09-16T04:43:07.702109+0000","last_clean":"2024-09-16T04:43:07.702109+0000","last_became_active":"2024-09-16T04:42:18.798872+0000","last_became_peered":"2024-09-16T04:42:18.798872+0000","last_unstale":"2024-09-16T04:43:07.702109+0000","last_undegraded":"2024-09-16T04:43:07.702109+0000","last_fullsized":"2024-09-16T04:43:07.702109+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:35.566994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.566994+0000","objects_scrubbed":6,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:43.854834+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0.414655147,"stat_sum":{"num_bytes":10485760,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":17,"num_write_kb":6864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.d","version":"105'203","reported_seq":577,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-09-16T04:43:08.696783+0000","last_change":"2024-09-16T04:42:57.825072+0000","last_active":"2024-09-16T04:43:08.696783+0000","last_peered":"2024-09-16T04:43:08.696783+0000","last_clean":"2024-09-16T04:43:08.696783+0000","last_became_active":"2024-09-16T04:42:19.196254+0000","last_became_peered":"2024-09-16T04:42:19.196254+0000","last_unstale":"2024-09-16T04:43:08.696783+0000","last_undegraded":"2024-09-16T04:43:08.696783+0000","last_fullsized":"2024-09-16T04:43:08.696783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'134","last_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_deep_scrub":"74'134","last_deep_scrub_stamp":"2024-09-16T04:42:25.664276+0000","last_clean_scrub_stamp":"2024-09-16T04:42:25.664276+0000","objects_scrubbed":9,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:53.596086+0000","scrub_duration":1071,"objects_trimmed":0,"snaptrim_duration":0.20199710000000001,"stat_sum":{"num_bytes":6995968,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":7,"num_read_kb":4921,"num_write":28,"num_write_kb":1936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.e","version":"92'88","reported_seq":479,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:09.704502+0000","last_change":"2024-09-16T04:42:57.725578+0000","last_active":"2024-09-16T04:43:09.704502+0000","last_peered":"2024-09-16T04:43:09.704502+0000","last_clean":"2024-09-16T04:43:09.704502+0000","last_became_active":"2024-09-16T04:42:18.700111+0000","last_became_peered":"2024-09-16T04:42:18.700111+0000","last_unstale":"2024-09-16T04:43:09.704502+0000","last_undegraded":"2024-09-16T04:43:09.704502+0000","last_fullsized":"2024-09-16T04:43:09.704502+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'79","last_scrub_stamp":"2024-09-16T04:42:34.646929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.646929+0000","objects_scrubbed":6,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.631984+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0.0021028940000000001,"stat_sum":{"num_bytes":5193728,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":9985,"num_write":1,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.f","version":"97'94","reported_seq":479,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T04:43:07.904084+0000","last_change":"2024-09-16T04:42:57.928111+0000","last_active":"2024-09-16T04:43:07.904084+0000","last_peered":"2024-09-16T04:43:07.904084+0000","last_clean":"2024-09-16T04:43:07.904084+0000","last_became_active":"2024-09-16T04:42:18.793319+0000","last_became_peered":"2024-09-16T04:42:18.793319+0000","last_unstale":"2024-09-16T04:43:07.904084+0000","last_undegraded":"2024-09-16T04:43:07.904084+0000","last_fullsized":"2024-09-16T04:43:07.904084+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'74","last_scrub_stamp":"2024-09-16T04:42:34.345034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:34.345034+0000","objects_scrubbed":7,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:53.597608+0000","scrub_duration":719,"objects_trimmed":0,"snaptrim_duration":0.20213098900000001,"stat_sum":{"num_bytes":3899392,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":12161,"num_write":5,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.10","version":"108'43","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.710073+0000","last_change":"2024-09-16T04:42:59.465896+0000","last_active":"2024-09-16T04:43:10.710073+0000","last_peered":"2024-09-16T04:43:10.710073+0000","last_clean":"2024-09-16T04:43:10.710073+0000","last_became_active":"2024-09-16T04:42:53.917288+0000","last_became_peered":"2024-09-16T04:42:53.917288+0000","last_unstale":"2024-09-16T04:43:10.710073+0000","last_undegraded":"2024-09-16T04:43:10.710073+0000","last_fullsized":"2024-09-16T04:43:10.710073+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":5,"last_scrub":"44'10","last_scrub_stamp":"2024-09-16T04:42:24.882865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:24.882865+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:43.863600+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0.396969236,"stat_sum":{"num_bytes":3162112,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":6,"num_read_kb":1682,"num_write":7,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3735552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,4,8,0,11],"acting":[13,6,7,4,8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.11","version":"107'60","reported_seq":398,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.070522+0000","last_change":"2024-09-16T04:42:58.030327+0000","last_active":"2024-09-16T04:43:10.070522+0000","last_peered":"2024-09-16T04:43:10.070522+0000","last_clean":"2024-09-16T04:43:10.070522+0000","last_became_active":"2024-09-16T04:42:53.807035+0000","last_became_peered":"2024-09-16T04:42:53.807035+0000","last_unstale":"2024-09-16T04:43:10.070522+0000","last_undegraded":"2024-09-16T04:43:10.070522+0000","last_fullsized":"2024-09-16T04:43:10.070522+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'28","last_scrub_stamp":"2024-09-16T04:42:10.158979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.158979+0000","objects_scrubbed":3,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:28.014899+0000","scrub_duration":855,"objects_trimmed":1,"snaptrim_duration":0.30489170799999998,"stat_sum":{"num_bytes":2179072,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5233,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2572288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,13,9,5,6,1],"acting":[10,11,13,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.12","version":"105'38","reported_seq":357,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T04:43:10.806563+0000","last_change":"2024-09-16T04:42:57.718276+0000","last_active":"2024-09-16T04:43:10.806563+0000","last_peered":"2024-09-16T04:43:10.806563+0000","last_clean":"2024-09-16T04:43:10.806563+0000","last_became_active":"2024-09-16T04:42:53.712428+0000","last_became_peered":"2024-09-16T04:42:53.712428+0000","last_unstale":"2024-09-16T04:43:10.806563+0000","last_undegraded":"2024-09-16T04:43:10.806563+0000","last_fullsized":"2024-09-16T04:43:10.806563+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'12","last_scrub_stamp":"2024-09-16T04:42:10.115913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.115913+0000","objects_scrubbed":1,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:22.191116+0000","scrub_duration":611,"objects_trimmed":0,"snaptrim_duration":0.10218153200000001,"stat_sum":{"num_bytes":2555904,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3745,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2555904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,4,3,12],"acting":[9,7,8,6,4,3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]},{"pgid":"3.13","version":"36'3","reported_seq":288,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T04:43:07.802660+0000","last_change":"2024-09-16T04:42:57.829230+0000","last_active":"2024-09-16T04:43:07.802660+0000","last_peered":"2024-09-16T04:43:07.802660+0000","last_clean":"2024-09-16T04:43:07.802660+0000","last_became_active":"2024-09-16T04:42:04.415210+0000","last_became_peered":"2024-09-16T04:42:04.415210+0000","last_unstale":"2024-09-16T04:43:07.802660+0000","last_undegraded":"2024-09-16T04:43:07.802660+0000","last_fullsized":"2024-09-16T04:43:07.802660+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"36'3","last_scrub_stamp":"2024-09-16T04:42:10.690368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:40.675693+0000","last_clean_scrub_stamp":"2024-09-16T04:42:10.690368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:43:23.944242+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0.10308851600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9,13,0,4,10],"acting":[11,7,9,13,0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"10"},{"start":"1a","length":"1"},{"start":"1c","length":"1"},{"start":"23","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":34,"stat_sum":{"num_bytes":115400704,"num_objects":90,"num_object_clones":46,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":7,"num_read":182,"num_read_kb":117853,"num_write":305,"num_write_kb":65360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":84115456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":247005184,"data_stored":302092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1751,"ondisk_log_size":1751,"up":238,"acting":238,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247834,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52364,"kb_used_data":17012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319476,"statfs":{"total":96636764160,"available":96583143424,"internally_reserved":0,"allocated":17420288,"data_stored":20964468,"data_compressed":313826,"data_compressed_allocated":425984,"data_compressed_original":1282048,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247834,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52216,"kb_used_data":16860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319624,"statfs":{"total":96636764160,"available":96583294976,"internally_reserved":0,"allocated":17264640,"data_stored":19429808,"data_compressed":284955,"data_compressed_allocated":389120,"data_compressed_original":778240,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247834,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56128,"kb_used_data":20776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315712,"statfs":{"total":96636764160,"available":96579289088,"internally_reserved":0,"allocated":21274624,"data_stored":24441428,"data_compressed":281881,"data_compressed_allocated":385024,"data_compressed_original":770048,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313240,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52704,"kb_used_data":17356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319136,"statfs":{"total":96636764160,"available":96582795264,"internally_reserved":0,"allocated":17772544,"data_stored":22346122,"data_compressed":272648,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313240,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50352,"kb_used_data":19164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321488,"statfs":{"total":96636764160,"available":96585203712,"internally_reserved":0,"allocated":19623936,"data_stored":23316874,"data_compressed":272648,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280537,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44504,"kb_used_data":13316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327336,"statfs":{"total":96636764160,"available":96591192064,"internally_reserved":0,"allocated":13635584,"data_stored":16589940,"data_compressed":313826,"data_compressed_allocated":425984,"data_compressed_original":1282048,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":0,"seq":2,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280537,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48412,"kb_used_data":17224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323428,"statfs":{"total":96636764160,"available":96587190272,"internally_reserved":0,"allocated":17637376,"data_stored":20584876,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53484,"kb_used_data":18136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318356,"statfs":{"total":96636764160,"available":96581996544,"internally_reserved":0,"allocated":18571264,"data_stored":21887385,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280537,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40880,"kb_used_data":9684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330960,"statfs":{"total":96636764160,"available":96594903040,"internally_reserved":0,"allocated":9916416,"data_stored":12888473,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280536,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48752,"kb_used_data":17564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323088,"statfs":{"total":96636764160,"available":96586842112,"internally_reserved":0,"allocated":17985536,"data_stored":20580761,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280537,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51372,"kb_used_data":16024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320468,"statfs":{"total":96636764160,"available":96584159232,"internally_reserved":0,"allocated":16408576,"data_stored":18057625,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280536,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60000,"kb_used_data":24644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94311840,"statfs":{"total":96636764160,"available":96575324160,"internally_reserved":0,"allocated":25235456,"data_stored":30493081,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280537,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47176,"kb_used_data":15988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324664,"statfs":{"total":96636764160,"available":96588455936,"internally_reserved":0,"allocated":16371712,"data_stored":21313945,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,6,7,8,9,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280536,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43560,"kb_used_data":12372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328280,"statfs":{"total":96636764160,"available":96592158720,"internally_reserved":0,"allocated":12668928,"data_stored":14272940,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280537,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49136,"kb_used_data":13788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322704,"statfs":{"total":96636764160,"available":96586448896,"internally_reserved":0,"allocated":14118912,"data_stored":17095065,"data_compressed":278814,"data_compressed_allocated":380928,"data_compressed_original":761856,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16453632,"data_stored":21344256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19918848,"data_stored":23408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12218368,"data_stored":15028224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16289792,"data_stored":19562496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17227776,"data_stored":20865024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8572928,"data_stored":11866112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16642048,"data_stored":19558400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15065088,"data_stored":17035264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":23891968,"data_stored":29470720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":15896576,"data_stored":18386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":15028224,"data_stored":20291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18305024,"data_stored":22315008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11321344,"data_stored":13250560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":16003072,"data_stored":19402752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12775424,"data_stored":16072704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":11395072,"data_stored":14233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:13.356 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T04:43:13.356 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T04:43:13.356 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-09-16T04:43:13.357 INFO:tasks.ceph.osd.14.smithi144.stderr:2024-09-16T04:43:13.343+0000 7fef70590640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26326) UID: 0 2024-09-16T04:43:13.357 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:43:13.357 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-09-16T04:43:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:43:13.370 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-09-16T04:43:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:43:13.391 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-09-16T04:43:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:43:13.412 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-09-16T04:43:13.412 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:43:13.430 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-09-16T04:43:13.430 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:43:13.448 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-09-16T04:43:13.448 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:43:13.449 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-09-16T04:43:13.446+0000 7ffad4932640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26295) UID: 0 2024-09-16T04:43:13.469 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-09-16T04:43:13.470 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:43:13.485 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:13.485 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:13.490 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-09-16T04:43:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:43:13.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:43:13.510 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-09-16T04:43:13.511 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:43:13.528 INFO:teuthology.orchestra.run.smithi059.stdout:osd.12: {} 2024-09-16T04:43:13.528 INFO:teuthology.orchestra.run.smithi059.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T04:43:13.549 INFO:teuthology.orchestra.run.smithi059.stdout:osd.13: {} 2024-09-16T04:43:13.549 INFO:teuthology.orchestra.run.smithi059.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T04:43:13.549 INFO:tasks.ceph.osd.12.smithi059.stderr:2024-09-16T04:43:13.542+0000 7fe031b7b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26362) UID: 0 2024-09-16T04:43:13.570 INFO:teuthology.orchestra.run.smithi059.stdout:osd.14: {} 2024-09-16T04:43:13.570 INFO:teuthology.orchestra.run.smithi059.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T04:43:13.589 INFO:teuthology.orchestra.run.smithi059.stdout:osd.15: {} 2024-09-16T04:43:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T04:43:13.625 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:13.625 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:13.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:43:13.650 INFO:tasks.ceph.osd.5.smithi139.stderr:2024-09-16T04:43:13.646+0000 7f901e8ea640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26298) UID: 0 2024-09-16T04:43:13.743 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'unique_pool_0' removedpool 'unique_pool_0' removed 2024-09-16T04:43:13.746 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:43:13.765 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:13.765 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:13.771 INFO:tasks.ceph.osd.8.smithi059.stderr:2024-09-16T04:43:13.766+0000 7f2a2e4b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26360) UID: 0 2024-09-16T04:43:13.771 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T04:43:13.781 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T04:43:13.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:43:13.906 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:13.906 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:13.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:43:14.054 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:14.054 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:14.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:43:14.193 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:14.194 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:14.753 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-09-16T04:43:15.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T04:43:15.968 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T04:43:15.969 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:43:15.982 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-16T04:43:16.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:43:17.759 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-09-16T04:43:17.789 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:43:18.762 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-09-16T04:43:20.983 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T04:43:20.983 INFO:tasks.thrashosds.thrasher:Adding osd 15 2024-09-16T04:43:20.983 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 15 2024-09-16T04:43:21.306 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-09-16T04:43:21.302+0000 7f2decafd640 -1 mon.a@0(leader).osd e123 definitely_dead 0 2024-09-16T04:43:22.094 INFO:teuthology.orchestra.run.smithi059.stderr:marked in osd.15. 2024-09-16T04:43:22.123 INFO:tasks.thrashosds.thrasher:Added osd 15 2024-09-16T04:43:22.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T04:43:22.265 INFO:teuthology.orchestra.run.smithi059.stdout:mon.a: {} 2024-09-16T04:43:22.265 INFO:teuthology.orchestra.run.smithi059.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:43:22.278 INFO:teuthology.orchestra.run.smithi059.stdout:mon.b: {} 2024-09-16T04:43:22.278 INFO:teuthology.orchestra.run.smithi059.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:43:22.293 INFO:teuthology.orchestra.run.smithi059.stdout:mon.c: {} 2024-09-16T04:43:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:43:22.310 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T04:43:23.097 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T04:43:23.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T04:43:24.107 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T04:43:24.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T04:43:25.117 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T04:43:25.146 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T04:43:26.127 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T04:43:26.154 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T04:43:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T04:43:27.163 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-16T04:43:28.155 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-16T04:43:28.178 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T04:43:29.157 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T04:43:29.186 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-16T04:43:30.171 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-16T04:43:30.195 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T04:43:31.191 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T04:43:31.219 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T04:43:32.193 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T04:43:32.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T04:43:33.203 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T04:43:33.227 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T04:43:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T04:43:34.244 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T04:43:35.226 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T04:43:35.251 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-16T04:43:36.244 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-16T04:43:36.267 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T04:43:37.247 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T04:43:37.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-16T04:43:38.258 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-16T04:43:38.286 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T04:43:39.268 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T04:43:39.295 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T04:43:40.278 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T04:43:40.306 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T04:43:41.288 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T04:43:41.314 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T04:43:42.301 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T04:43:42.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T04:43:43.321 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T04:43:43.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-16T04:43:44.324 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-16T04:43:44.349 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T04:43:45.342 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T04:43:45.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-16T04:43:46.345 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-16T04:43:46.369 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T04:43:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T04:43:47.388 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T04:43:48.363 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T04:43:48.388 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T04:43:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T04:43:49.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T04:43:50.385 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T04:43:50.413 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T04:43:51.400 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T04:43:51.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-16T04:43:52.403 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-16T04:43:52.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T04:43:53.414 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T04:43:53.439 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-16T04:43:54.425 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-16T04:43:54.455 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T04:43:54.455 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T04:43:54.455 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T04:43:54.455 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T04:43:54.455 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:43:54.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:43:54.757 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:54.757 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":156,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:43:54.421149+0000","last_up_change":"2024-09-16T04:43:12.718474+0000","last_in_change":"2024-09-16T04:43:22.084893+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:41:23.074365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:41:25.522114+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6801","nonce":1228226348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6803","nonce":1228226348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6807","nonce":1228226348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6805","nonce":1228226348}]},"public_addr":"172.21.15.59:6801/1228226348","cluster_addr":"172.21.15.59:6803/1228226348","heartbeat_back_addr":"172.21.15.59:6807/1228226348","heartbeat_front_addr":"172.21.15.59:6805/1228226348","state":["exists","up"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6817","nonce":2731115256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6819","nonce":2731115256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6823","nonce":2731115256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6821","nonce":2731115256}]},"public_addr":"172.21.15.139:6817/2731115256","cluster_addr":"172.21.15.139:6819/2731115256","heartbeat_back_addr":"172.21.15.139:6823/2731115256","heartbeat_front_addr":"172.21.15.139:6821/2731115256","state":["exists","up"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6825","nonce":76637888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6827","nonce":76637888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6831","nonce":76637888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6829","nonce":76637888}]},"public_addr":"172.21.15.144:6825/76637888","cluster_addr":"172.21.15.144:6827/76637888","heartbeat_back_addr":"172.21.15.144:6831/76637888","heartbeat_front_addr":"172.21.15.144:6829/76637888","state":["exists","up"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6803","nonce":58455506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6805","nonce":58455506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6809","nonce":58455506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6807","nonce":58455506}]},"public_addr":"172.21.15.165:6803/58455506","cluster_addr":"172.21.15.165:6805/58455506","heartbeat_back_addr":"172.21.15.165:6809/58455506","heartbeat_front_addr":"172.21.15.165:6807/58455506","state":["exists","up"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6825","nonce":2712399685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6827","nonce":2712399685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6831","nonce":2712399685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6829","nonce":2712399685}]},"public_addr":"172.21.15.59:6825/2712399685","cluster_addr":"172.21.15.59:6827/2712399685","heartbeat_back_addr":"172.21.15.59:6831/2712399685","heartbeat_front_addr":"172.21.15.59:6829/2712399685","state":["exists","up"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6801","nonce":2946170755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6803","nonce":2946170755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6807","nonce":2946170755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6805","nonce":2946170755}]},"public_addr":"172.21.15.139:6801/2946170755","cluster_addr":"172.21.15.139:6803/2946170755","heartbeat_back_addr":"172.21.15.139:6807/2946170755","heartbeat_front_addr":"172.21.15.139:6805/2946170755","state":["exists","up"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6801","nonce":3812816248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6803","nonce":3812816248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6807","nonce":3812816248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6805","nonce":3812816248}]},"public_addr":"172.21.15.144:6801/3812816248","cluster_addr":"172.21.15.144:6803/3812816248","heartbeat_back_addr":"172.21.15.144:6807/3812816248","heartbeat_front_addr":"172.21.15.144:6805/3812816248","state":["exists","up"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6819","nonce":1352862134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6821","nonce":1352862134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6825","nonce":1352862134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6823","nonce":1352862134}]},"public_addr":"172.21.15.165:6819/1352862134","cluster_addr":"172.21.15.165:6821/1352862134","heartbeat_back_addr":"172.21.15.165:6825/1352862134","heartbeat_front_addr":"172.21.15.165:6823/1352862134","state":["exists","up"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6817","nonce":1451673997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6819","nonce":1451673997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6823","nonce":1451673997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6821","nonce":1451673997}]},"public_addr":"172.21.15.59:6817/1451673997","cluster_addr":"172.21.15.59:6819/1451673997","heartbeat_back_addr":"172.21.15.59:6823/1451673997","heartbeat_front_addr":"172.21.15.59:6821/1451673997","state":["exists","up"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6809","nonce":174106571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6811","nonce":174106571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6815","nonce":174106571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6813","nonce":174106571}]},"public_addr":"172.21.15.139:6809/174106571","cluster_addr":"172.21.15.139:6811/174106571","heartbeat_back_addr":"172.21.15.139:6815/174106571","heartbeat_front_addr":"172.21.15.139:6813/174106571","state":["exists","up"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6809","nonce":66779937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6812","nonce":66779937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6818","nonce":66779937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6815","nonce":66779937}]},"public_addr":"172.21.15.144:6809/66779937","cluster_addr":"172.21.15.144:6812/66779937","heartbeat_back_addr":"172.21.15.144:6818/66779937","heartbeat_front_addr":"172.21.15.144:6815/66779937","state":["exists","up"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6811","nonce":49204916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6813","nonce":49204916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6817","nonce":49204916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6815","nonce":49204916}]},"public_addr":"172.21.15.165:6811/49204916","cluster_addr":"172.21.15.165:6813/49204916","heartbeat_back_addr":"172.21.15.165:6817/49204916","heartbeat_front_addr":"172.21.15.165:6815/49204916","state":["exists","up"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6809","nonce":3097346146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6811","nonce":3097346146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6815","nonce":3097346146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6813","nonce":3097346146}]},"public_addr":"172.21.15.59:6809/3097346146","cluster_addr":"172.21.15.59:6811/3097346146","heartbeat_back_addr":"172.21.15.59:6815/3097346146","heartbeat_front_addr":"172.21.15.59:6813/3097346146","state":["exists","up"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6825","nonce":2903817823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6827","nonce":2903817823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6831","nonce":2903817823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6829","nonce":2903817823}]},"public_addr":"172.21.15.139:6825/2903817823","cluster_addr":"172.21.15.139:6827/2903817823","heartbeat_back_addr":"172.21.15.139:6831/2903817823","heartbeat_front_addr":"172.21.15.139:6829/2903817823","state":["exists","up"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6811","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6814","nonce":229023981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6819","nonce":229023981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6823","nonce":229023981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6821","nonce":229023981}]},"public_addr":"172.21.15.144:6814/229023981","cluster_addr":"172.21.15.144:6819/229023981","heartbeat_back_addr":"172.21.15.144:6823/229023981","heartbeat_front_addr":"172.21.15.144:6821/229023981","state":["exists","up"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":99,"up_from":119,"up_thru":124,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6827","nonce":2616552093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6829","nonce":2616552093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6833","nonce":2616552093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6831","nonce":2616552093}]},"public_addr":"172.21.15.165:6827/2616552093","cluster_addr":"172.21.15.165:6829/2616552093","heartbeat_back_addr":"172.21.15.165:6833/2616552093","heartbeat_front_addr":"172.21.15.165:6831/2616552093","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:48.826878+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:40.740589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:44.548804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:53.306808+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:42.924457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:47.648804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:49.832912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:48.874174+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:38.827784+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:30.085848+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:37.768155+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:41.884656+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:48.659730+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:51.158730+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:52.799239+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:12.392064+0000","dead_epoch":101}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3,"length":1},{"begin":7,"length":1},{"begin":9,"length":16},{"begin":26,"length":1},{"begin":28,"length":1},{"begin":35,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:43:54.775 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:43:54.775 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:43:55.084 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:55.084 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":156,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:43:54.421149+0000","last_up_change":"2024-09-16T04:43:12.718474+0000","last_in_change":"2024-09-16T04:43:22.084893+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:41:23.074365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:41:25.522114+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6801","nonce":1228226348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6803","nonce":1228226348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6807","nonce":1228226348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6805","nonce":1228226348}]},"public_addr":"172.21.15.59:6801/1228226348","cluster_addr":"172.21.15.59:6803/1228226348","heartbeat_back_addr":"172.21.15.59:6807/1228226348","heartbeat_front_addr":"172.21.15.59:6805/1228226348","state":["exists","up"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6817","nonce":2731115256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6819","nonce":2731115256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6823","nonce":2731115256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6821","nonce":2731115256}]},"public_addr":"172.21.15.139:6817/2731115256","cluster_addr":"172.21.15.139:6819/2731115256","heartbeat_back_addr":"172.21.15.139:6823/2731115256","heartbeat_front_addr":"172.21.15.139:6821/2731115256","state":["exists","up"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6825","nonce":76637888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6827","nonce":76637888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6831","nonce":76637888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6829","nonce":76637888}]},"public_addr":"172.21.15.144:6825/76637888","cluster_addr":"172.21.15.144:6827/76637888","heartbeat_back_addr":"172.21.15.144:6831/76637888","heartbeat_front_addr":"172.21.15.144:6829/76637888","state":["exists","up"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6803","nonce":58455506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6805","nonce":58455506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6809","nonce":58455506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6807","nonce":58455506}]},"public_addr":"172.21.15.165:6803/58455506","cluster_addr":"172.21.15.165:6805/58455506","heartbeat_back_addr":"172.21.15.165:6809/58455506","heartbeat_front_addr":"172.21.15.165:6807/58455506","state":["exists","up"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6825","nonce":2712399685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6827","nonce":2712399685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6831","nonce":2712399685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6829","nonce":2712399685}]},"public_addr":"172.21.15.59:6825/2712399685","cluster_addr":"172.21.15.59:6827/2712399685","heartbeat_back_addr":"172.21.15.59:6831/2712399685","heartbeat_front_addr":"172.21.15.59:6829/2712399685","state":["exists","up"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6801","nonce":2946170755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6803","nonce":2946170755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6807","nonce":2946170755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6805","nonce":2946170755}]},"public_addr":"172.21.15.139:6801/2946170755","cluster_addr":"172.21.15.139:6803/2946170755","heartbeat_back_addr":"172.21.15.139:6807/2946170755","heartbeat_front_addr":"172.21.15.139:6805/2946170755","state":["exists","up"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6801","nonce":3812816248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6803","nonce":3812816248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6807","nonce":3812816248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6805","nonce":3812816248}]},"public_addr":"172.21.15.144:6801/3812816248","cluster_addr":"172.21.15.144:6803/3812816248","heartbeat_back_addr":"172.21.15.144:6807/3812816248","heartbeat_front_addr":"172.21.15.144:6805/3812816248","state":["exists","up"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6819","nonce":1352862134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6821","nonce":1352862134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6825","nonce":1352862134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6823","nonce":1352862134}]},"public_addr":"172.21.15.165:6819/1352862134","cluster_addr":"172.21.15.165:6821/1352862134","heartbeat_back_addr":"172.21.15.165:6825/1352862134","heartbeat_front_addr":"172.21.15.165:6823/1352862134","state":["exists","up"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6817","nonce":1451673997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6819","nonce":1451673997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6823","nonce":1451673997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6821","nonce":1451673997}]},"public_addr":"172.21.15.59:6817/1451673997","cluster_addr":"172.21.15.59:6819/1451673997","heartbeat_back_addr":"172.21.15.59:6823/1451673997","heartbeat_front_addr":"172.21.15.59:6821/1451673997","state":["exists","up"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6809","nonce":174106571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6811","nonce":174106571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6815","nonce":174106571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6813","nonce":174106571}]},"public_addr":"172.21.15.139:6809/174106571","cluster_addr":"172.21.15.139:6811/174106571","heartbeat_back_addr":"172.21.15.139:6815/174106571","heartbeat_front_addr":"172.21.15.139:6813/174106571","state":["exists","up"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6809","nonce":66779937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6812","nonce":66779937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6818","nonce":66779937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6815","nonce":66779937}]},"public_addr":"172.21.15.144:6809/66779937","cluster_addr":"172.21.15.144:6812/66779937","heartbeat_back_addr":"172.21.15.144:6818/66779937","heartbeat_front_addr":"172.21.15.144:6815/66779937","state":["exists","up"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6811","nonce":49204916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6813","nonce":49204916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6817","nonce":49204916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6815","nonce":49204916}]},"public_addr":"172.21.15.165:6811/49204916","cluster_addr":"172.21.15.165:6813/49204916","heartbeat_back_addr":"172.21.15.165:6817/49204916","heartbeat_front_addr":"172.21.15.165:6815/49204916","state":["exists","up"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6809","nonce":3097346146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6811","nonce":3097346146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6815","nonce":3097346146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6813","nonce":3097346146}]},"public_addr":"172.21.15.59:6809/3097346146","cluster_addr":"172.21.15.59:6811/3097346146","heartbeat_back_addr":"172.21.15.59:6815/3097346146","heartbeat_front_addr":"172.21.15.59:6813/3097346146","state":["exists","up"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6825","nonce":2903817823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6827","nonce":2903817823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6831","nonce":2903817823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6829","nonce":2903817823}]},"public_addr":"172.21.15.139:6825/2903817823","cluster_addr":"172.21.15.139:6827/2903817823","heartbeat_back_addr":"172.21.15.139:6831/2903817823","heartbeat_front_addr":"172.21.15.139:6829/2903817823","state":["exists","up"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6811","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6814","nonce":229023981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6819","nonce":229023981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6823","nonce":229023981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6821","nonce":229023981}]},"public_addr":"172.21.15.144:6814/229023981","cluster_addr":"172.21.15.144:6819/229023981","heartbeat_back_addr":"172.21.15.144:6823/229023981","heartbeat_front_addr":"172.21.15.144:6821/229023981","state":["exists","up"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":99,"up_from":119,"up_thru":124,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6827","nonce":2616552093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6829","nonce":2616552093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6833","nonce":2616552093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6831","nonce":2616552093}]},"public_addr":"172.21.15.165:6827/2616552093","cluster_addr":"172.21.15.165:6829/2616552093","heartbeat_back_addr":"172.21.15.165:6833/2616552093","heartbeat_front_addr":"172.21.15.165:6831/2616552093","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:48.826878+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:40.740589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:44.548804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:53.306808+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:42.924457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:47.648804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:49.832912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:48.874174+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:38.827784+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:30.085848+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:37.768155+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:41.884656+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:48.659730+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:51.158730+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:52.799239+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:12.392064+0000","dead_epoch":101}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3,"length":1},{"begin":7,"length":1},{"begin":9,"length":16},{"begin":26,"length":1},{"begin":28,"length":1},{"begin":35,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:43:55.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:43:55.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:43:55.106 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:43:55.106 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:43:55.107 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:43:55.107 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:43:55.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T04:43:55.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T04:43:55.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T04:43:55.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T04:43:55.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T04:43:55.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T04:43:55.111 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T04:43:55.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T04:43:55.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T04:43:55.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T04:43:55.533 INFO:teuthology.orchestra.run.smithi059.stdout:98784247843 2024-09-16T04:43:55.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:43:55.537 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:55.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T04:43:55.548 INFO:teuthology.orchestra.run.smithi059.stdout:511101108235 2024-09-16T04:43:55.548 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T04:43:55.561 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:55.561 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:43:55.615 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:55.615 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:43:55.623 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:55.624 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:43:55.625 INFO:teuthology.orchestra.run.smithi059.stdout:90194313250 2024-09-16T04:43:55.625 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:43:55.628 INFO:teuthology.orchestra.run.smithi059.stdout:98784247843 2024-09-16T04:43:55.628 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:43:55.652 INFO:teuthology.orchestra.run.smithi059.stdout:90194313250 2024-09-16T04:43:55.652 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:43:55.708 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:55.708 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:43:55.717 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:55.717 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:43:55.721 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:55.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T04:43:55.744 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:55.744 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:43:55.745 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:55.745 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:43:55.770 INFO:teuthology.orchestra.run.smithi059.stdout:98784247843 2024-09-16T04:43:55.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T04:43:55.797 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:55.798 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:43:56.321 INFO:teuthology.orchestra.run.smithi059.stdout:98784247842 2024-09-16T04:43:56.367 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247843 got 98784247842 for osd.1 2024-09-16T04:43:56.394 INFO:teuthology.orchestra.run.smithi059.stdout:511101108234 2024-09-16T04:43:56.434 INFO:teuthology.orchestra.run.smithi059.stdout:98784247842 2024-09-16T04:43:56.450 INFO:teuthology.orchestra.run.smithi059.stdout:94489280545 2024-09-16T04:43:56.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 511101108235 got 511101108234 for osd.15 2024-09-16T04:43:56.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247843 got 98784247842 for osd.9 2024-09-16T04:43:56.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280545 for osd.12 2024-09-16T04:43:56.550 INFO:teuthology.orchestra.run.smithi059.stdout:94489280545 2024-09-16T04:43:56.583 INFO:teuthology.orchestra.run.smithi059.stdout:94489280545 2024-09-16T04:43:56.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280545 for osd.6 2024-09-16T04:43:56.617 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:56.635 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280545 for osd.2 2024-09-16T04:43:56.650 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:56.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280546 for osd.14 2024-09-16T04:43:56.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280546 for osd.4 2024-09-16T04:43:56.693 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:56.707 INFO:teuthology.orchestra.run.smithi059.stdout:90194313249 2024-09-16T04:43:56.720 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280546 for osd.5 2024-09-16T04:43:56.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313250 got 90194313249 for osd.0 2024-09-16T04:43:56.738 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:56.756 INFO:teuthology.orchestra.run.smithi059.stdout:98784247842 2024-09-16T04:43:56.756 INFO:teuthology.orchestra.run.smithi059.stdout:90194313249 2024-09-16T04:43:56.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280546 for osd.3 2024-09-16T04:43:56.769 INFO:teuthology.orchestra.run.smithi059.stdout:94489280545 2024-09-16T04:43:56.774 INFO:teuthology.orchestra.run.smithi059.stdout:94489280545 2024-09-16T04:43:56.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313250 got 90194313249 for osd.11 2024-09-16T04:43:56.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247843 got 98784247842 for osd.13 2024-09-16T04:43:56.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280545 for osd.8 2024-09-16T04:43:56.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280545 for osd.10 2024-09-16T04:43:56.836 INFO:teuthology.orchestra.run.smithi059.stdout:94489280545 2024-09-16T04:43:56.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280545 for osd.7 2024-09-16T04:43:57.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:43:57.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T04:43:57.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:43:57.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T04:43:57.593 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:43:57.636 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:43:57.665 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T04:43:57.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:43:57.697 INFO:teuthology.orchestra.run.smithi059.stdout:98784247844 2024-09-16T04:43:57.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:43:57.727 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247843 got 98784247844 for osd.1 2024-09-16T04:43:57.727 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:57.733 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:43:57.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:43:57.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:43:57.787 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T04:43:57.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:43:57.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:43:57.855 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:43:57.899 INFO:teuthology.orchestra.run.smithi059.stdout:511101108235 2024-09-16T04:43:57.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 511101108235 got 511101108235 for osd.15 2024-09-16T04:43:57.976 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.014 INFO:teuthology.orchestra.run.smithi059.stdout:98784247844 2024-09-16T04:43:58.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247843 got 98784247844 for osd.9 2024-09-16T04:43:58.042 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.097 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:58.155 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280546 for osd.12 2024-09-16T04:43:58.155 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.174 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:58.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280546 for osd.6 2024-09-16T04:43:58.217 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.338 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:58.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280547 for osd.2 2024-09-16T04:43:58.398 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.403 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:58.406 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:58.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280547 for osd.14 2024-09-16T04:43:58.464 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.466 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:58.468 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280547 for osd.4 2024-09-16T04:43:58.468 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.502 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280547 for osd.5 2024-09-16T04:43:58.502 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.517 INFO:teuthology.orchestra.run.smithi059.stdout:90194313250 2024-09-16T04:43:58.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313250 got 90194313250 for osd.0 2024-09-16T04:43:58.545 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.563 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:58.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280547 for osd.3 2024-09-16T04:43:58.588 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.590 INFO:teuthology.orchestra.run.smithi059.stdout:98784247844 2024-09-16T04:43:58.595 INFO:teuthology.orchestra.run.smithi059.stdout:90194313250 2024-09-16T04:43:58.596 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:58.612 INFO:teuthology.orchestra.run.smithi059.stdout:94489280546 2024-09-16T04:43:58.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313250 got 90194313250 for osd.11 2024-09-16T04:43:58.618 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280547 for osd.10 2024-09-16T04:43:58.620 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.623 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247843 got 98784247844 for osd.13 2024-09-16T04:43:58.623 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280546 for osd.8 2024-09-16T04:43:58.632 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.660 INFO:teuthology.orchestra.run.smithi059.stdout:94489280547 2024-09-16T04:43:58.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280547 got 94489280547 for osd.7 2024-09-16T04:43:58.677 DEBUG:teuthology.parallel:result is None 2024-09-16T04:43:58.677 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T04:43:58.677 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:59.009 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:59.009 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:43:59.026 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-16T04:43:57.043240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":557272,"kb_used_data":29080,"kb_used_omap":27,"kb_used_meta":527780,"kb_avail":1509392168,"statfs":{"total":1546188226560,"available":1545617580032,"internally_reserved":0,"allocated":29777920,"data_stored":23865214,"data_compressed":6513500,"data_compressed_allocated":8871936,"data_compressed_original":18604032,"omap_allocated":28617,"internal_metadata":540446775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.636509"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":69,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.677123+0000","last_change":"2024-09-16T04:43:23.308987+0000","last_active":"2024-09-16T04:43:54.677123+0000","last_peered":"2024-09-16T04:43:54.677123+0000","last_clean":"2024-09-16T04:43:54.677123+0000","last_became_active":"2024-09-16T04:43:23.208392+0000","last_became_peered":"2024-09-16T04:43:23.208392+0000","last_unstale":"2024-09-16T04:43:54.677123+0000","last_undegraded":"2024-09-16T04:43:54.677123+0000","last_fullsized":"2024-09-16T04:43:54.677123+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:07.865595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":281,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.654448+0000","last_change":"2024-09-16T04:43:13.855573+0000","last_active":"2024-09-16T04:43:55.654448+0000","last_peered":"2024-09-16T04:43:55.654448+0000","last_clean":"2024-09-16T04:43:55.654448+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:43:55.654448+0000","last_undegraded":"2024-09-16T04:43:55.654448+0000","last_fullsized":"2024-09-16T04:43:55.654448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.855454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:18.040654+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":283,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555167+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:43:55.555167+0000","last_peered":"2024-09-16T04:43:55.555167+0000","last_clean":"2024-09-16T04:43:55.555167+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:43:55.555167+0000","last_undegraded":"2024-09-16T04:43:55.555167+0000","last_fullsized":"2024-09-16T04:43:55.555167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":281,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.452532+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:43:54.452532+0000","last_peered":"2024-09-16T04:43:54.452532+0000","last_clean":"2024-09-16T04:43:54.452532+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:43:54.452532+0000","last_undegraded":"2024-09-16T04:43:54.452532+0000","last_fullsized":"2024-09-16T04:43:54.452532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":291,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585320+0000","last_change":"2024-09-16T04:43:50.333177+0000","last_active":"2024-09-16T04:43:54.585320+0000","last_peered":"2024-09-16T04:43:54.585320+0000","last_clean":"2024-09-16T04:43:54.585320+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:43:54.585320+0000","last_undegraded":"2024-09-16T04:43:54.585320+0000","last_fullsized":"2024-09-16T04:43:54.585320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:43:50.332925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:50.332925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:50.759622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":284,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.656155+0000","last_change":"2024-09-16T04:43:13.999043+0000","last_active":"2024-09-16T04:43:55.656155+0000","last_peered":"2024-09-16T04:43:55.656155+0000","last_clean":"2024-09-16T04:43:55.656155+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:43:55.656155+0000","last_undegraded":"2024-09-16T04:43:55.656155+0000","last_fullsized":"2024-09-16T04:43:55.656155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.998660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.998660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.784972+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":285,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585283+0000","last_change":"2024-09-16T04:43:22.549838+0000","last_active":"2024-09-16T04:43:54.585283+0000","last_peered":"2024-09-16T04:43:54.585283+0000","last_clean":"2024-09-16T04:43:54.585283+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:43:54.585283+0000","last_undegraded":"2024-09-16T04:43:54.585283+0000","last_fullsized":"2024-09-16T04:43:54.585283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:22.549596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:22.549596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:29.819458+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"111'2","reported_seq":304,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555236+0000","last_change":"2024-09-16T04:43:23.813052+0000","last_active":"2024-09-16T04:43:55.555236+0000","last_peered":"2024-09-16T04:43:55.555236+0000","last_clean":"2024-09-16T04:43:55.555236+0000","last_became_active":"2024-09-16T04:43:23.306721+0000","last_became_peered":"2024-09-16T04:43:23.306721+0000","last_unstale":"2024-09-16T04:43:55.555236+0000","last_undegraded":"2024-09-16T04:43:55.555236+0000","last_fullsized":"2024-09-16T04:43:55.555236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.979750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:24.154181+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":486,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.756267+0000","last_change":"2024-09-16T04:43:39.335810+0000","last_active":"2024-09-16T04:43:55.756267+0000","last_peered":"2024-09-16T04:43:55.756267+0000","last_clean":"2024-09-16T04:43:55.756267+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:43:55.756267+0000","last_undegraded":"2024-09-16T04:43:55.756267+0000","last_fullsized":"2024-09-16T04:43:55.756267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:43:39.335522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:43:39.335522+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:44.603174+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37164,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334676,"statfs":{"total":96636764160,"available":96598708224,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":119,"seq":511101108235,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28844,"kb_used_data":1808,"kb_used_omap":4,"kb_used_meta":27003,"kb_avail":94342996,"statfs":{"total":96636764160,"available":96607227904,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":4689,"internal_metadata":27651503},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:59.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:59.308 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:59.309 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:43:59.326 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T04:43:59.043919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":557312,"kb_used_data":29116,"kb_used_omap":27,"kb_used_meta":527780,"kb_avail":1509392128,"statfs":{"total":1546188226560,"available":1545617539072,"internally_reserved":0,"allocated":29814784,"data_stored":23894338,"data_compressed":6522539,"data_compressed_allocated":8884224,"data_compressed_original":18628608,"omap_allocated":28617,"internal_metadata":540446775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001993"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.971837+0000","last_change":"2024-09-16T04:43:23.308987+0000","last_active":"2024-09-16T04:43:55.971837+0000","last_peered":"2024-09-16T04:43:55.971837+0000","last_clean":"2024-09-16T04:43:55.971837+0000","last_became_active":"2024-09-16T04:43:23.208392+0000","last_became_peered":"2024-09-16T04:43:23.208392+0000","last_unstale":"2024-09-16T04:43:55.971837+0000","last_undegraded":"2024-09-16T04:43:55.971837+0000","last_fullsized":"2024-09-16T04:43:55.971837+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:07.865595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":281,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.654448+0000","last_change":"2024-09-16T04:43:13.855573+0000","last_active":"2024-09-16T04:43:55.654448+0000","last_peered":"2024-09-16T04:43:55.654448+0000","last_clean":"2024-09-16T04:43:55.654448+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:43:55.654448+0000","last_undegraded":"2024-09-16T04:43:55.654448+0000","last_fullsized":"2024-09-16T04:43:55.654448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.855454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:18.040654+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":283,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555167+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:43:55.555167+0000","last_peered":"2024-09-16T04:43:55.555167+0000","last_clean":"2024-09-16T04:43:55.555167+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:43:55.555167+0000","last_undegraded":"2024-09-16T04:43:55.555167+0000","last_fullsized":"2024-09-16T04:43:55.555167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":281,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.452532+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:43:54.452532+0000","last_peered":"2024-09-16T04:43:54.452532+0000","last_clean":"2024-09-16T04:43:54.452532+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:43:54.452532+0000","last_undegraded":"2024-09-16T04:43:54.452532+0000","last_fullsized":"2024-09-16T04:43:54.452532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":291,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585320+0000","last_change":"2024-09-16T04:43:50.333177+0000","last_active":"2024-09-16T04:43:54.585320+0000","last_peered":"2024-09-16T04:43:54.585320+0000","last_clean":"2024-09-16T04:43:54.585320+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:43:54.585320+0000","last_undegraded":"2024-09-16T04:43:54.585320+0000","last_fullsized":"2024-09-16T04:43:54.585320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:43:50.332925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:50.332925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:50.759622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":284,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.656155+0000","last_change":"2024-09-16T04:43:13.999043+0000","last_active":"2024-09-16T04:43:55.656155+0000","last_peered":"2024-09-16T04:43:55.656155+0000","last_clean":"2024-09-16T04:43:55.656155+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:43:55.656155+0000","last_undegraded":"2024-09-16T04:43:55.656155+0000","last_fullsized":"2024-09-16T04:43:55.656155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.998660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.998660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.784972+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":285,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585283+0000","last_change":"2024-09-16T04:43:22.549838+0000","last_active":"2024-09-16T04:43:54.585283+0000","last_peered":"2024-09-16T04:43:54.585283+0000","last_clean":"2024-09-16T04:43:54.585283+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:43:54.585283+0000","last_undegraded":"2024-09-16T04:43:54.585283+0000","last_fullsized":"2024-09-16T04:43:54.585283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:22.549596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:22.549596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:29.819458+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"111'2","reported_seq":304,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555236+0000","last_change":"2024-09-16T04:43:23.813052+0000","last_active":"2024-09-16T04:43:55.555236+0000","last_peered":"2024-09-16T04:43:55.555236+0000","last_clean":"2024-09-16T04:43:55.555236+0000","last_became_active":"2024-09-16T04:43:23.306721+0000","last_became_peered":"2024-09-16T04:43:23.306721+0000","last_unstale":"2024-09-16T04:43:55.555236+0000","last_undegraded":"2024-09-16T04:43:55.555236+0000","last_fullsized":"2024-09-16T04:43:55.555236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.979750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:24.154181+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":486,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.756267+0000","last_change":"2024-09-16T04:43:39.335810+0000","last_active":"2024-09-16T04:43:55.756267+0000","last_peered":"2024-09-16T04:43:55.756267+0000","last_clean":"2024-09-16T04:43:55.756267+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:43:55.756267+0000","last_undegraded":"2024-09-16T04:43:55.756267+0000","last_fullsized":"2024-09-16T04:43:55.756267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:43:39.335522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:43:39.335522+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:44.603174+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37164,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334676,"statfs":{"total":96636764160,"available":96598708224,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33016,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":119,"seq":511101108236,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28860,"kb_used_data":1820,"kb_used_omap":4,"kb_used_meta":27003,"kb_avail":94342980,"statfs":{"total":96636764160,"available":96607211520,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":4689,"internal_metadata":27651503},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:59.327 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T04:43:59.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T04:43:59.638 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:43:59.647 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:43:59.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:43:59.942 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:43:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:43:59.959 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T04:43:59.043919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":557312,"kb_used_data":29116,"kb_used_omap":27,"kb_used_meta":527780,"kb_avail":1509392128,"statfs":{"total":1546188226560,"available":1545617539072,"internally_reserved":0,"allocated":29814784,"data_stored":23894338,"data_compressed":6522539,"data_compressed_allocated":8884224,"data_compressed_original":18628608,"omap_allocated":28617,"internal_metadata":540446775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001993"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.971837+0000","last_change":"2024-09-16T04:43:23.308987+0000","last_active":"2024-09-16T04:43:55.971837+0000","last_peered":"2024-09-16T04:43:55.971837+0000","last_clean":"2024-09-16T04:43:55.971837+0000","last_became_active":"2024-09-16T04:43:23.208392+0000","last_became_peered":"2024-09-16T04:43:23.208392+0000","last_unstale":"2024-09-16T04:43:55.971837+0000","last_undegraded":"2024-09-16T04:43:55.971837+0000","last_fullsized":"2024-09-16T04:43:55.971837+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:07.865595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":281,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.654448+0000","last_change":"2024-09-16T04:43:13.855573+0000","last_active":"2024-09-16T04:43:55.654448+0000","last_peered":"2024-09-16T04:43:55.654448+0000","last_clean":"2024-09-16T04:43:55.654448+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:43:55.654448+0000","last_undegraded":"2024-09-16T04:43:55.654448+0000","last_fullsized":"2024-09-16T04:43:55.654448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.855454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:18.040654+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":283,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555167+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:43:55.555167+0000","last_peered":"2024-09-16T04:43:55.555167+0000","last_clean":"2024-09-16T04:43:55.555167+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:43:55.555167+0000","last_undegraded":"2024-09-16T04:43:55.555167+0000","last_fullsized":"2024-09-16T04:43:55.555167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":281,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.452532+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:43:54.452532+0000","last_peered":"2024-09-16T04:43:54.452532+0000","last_clean":"2024-09-16T04:43:54.452532+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:43:54.452532+0000","last_undegraded":"2024-09-16T04:43:54.452532+0000","last_fullsized":"2024-09-16T04:43:54.452532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":291,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585320+0000","last_change":"2024-09-16T04:43:50.333177+0000","last_active":"2024-09-16T04:43:54.585320+0000","last_peered":"2024-09-16T04:43:54.585320+0000","last_clean":"2024-09-16T04:43:54.585320+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:43:54.585320+0000","last_undegraded":"2024-09-16T04:43:54.585320+0000","last_fullsized":"2024-09-16T04:43:54.585320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:43:50.332925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:50.332925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:50.759622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":284,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.656155+0000","last_change":"2024-09-16T04:43:13.999043+0000","last_active":"2024-09-16T04:43:55.656155+0000","last_peered":"2024-09-16T04:43:55.656155+0000","last_clean":"2024-09-16T04:43:55.656155+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:43:55.656155+0000","last_undegraded":"2024-09-16T04:43:55.656155+0000","last_fullsized":"2024-09-16T04:43:55.656155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.998660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.998660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.784972+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":285,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585283+0000","last_change":"2024-09-16T04:43:22.549838+0000","last_active":"2024-09-16T04:43:54.585283+0000","last_peered":"2024-09-16T04:43:54.585283+0000","last_clean":"2024-09-16T04:43:54.585283+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:43:54.585283+0000","last_undegraded":"2024-09-16T04:43:54.585283+0000","last_fullsized":"2024-09-16T04:43:54.585283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:22.549596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:22.549596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:29.819458+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"111'2","reported_seq":304,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555236+0000","last_change":"2024-09-16T04:43:23.813052+0000","last_active":"2024-09-16T04:43:55.555236+0000","last_peered":"2024-09-16T04:43:55.555236+0000","last_clean":"2024-09-16T04:43:55.555236+0000","last_became_active":"2024-09-16T04:43:23.306721+0000","last_became_peered":"2024-09-16T04:43:23.306721+0000","last_unstale":"2024-09-16T04:43:55.555236+0000","last_undegraded":"2024-09-16T04:43:55.555236+0000","last_fullsized":"2024-09-16T04:43:55.555236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.979750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:24.154181+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":486,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.756267+0000","last_change":"2024-09-16T04:43:39.335810+0000","last_active":"2024-09-16T04:43:55.756267+0000","last_peered":"2024-09-16T04:43:55.756267+0000","last_clean":"2024-09-16T04:43:55.756267+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:43:55.756267+0000","last_undegraded":"2024-09-16T04:43:55.756267+0000","last_fullsized":"2024-09-16T04:43:55.756267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:43:39.335522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:43:39.335522+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:44.603174+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37164,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334676,"statfs":{"total":96636764160,"available":96598708224,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33016,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":119,"seq":511101108236,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28860,"kb_used_data":1820,"kb_used_omap":4,"kb_used_meta":27003,"kb_avail":94342980,"statfs":{"total":96636764160,"available":96607211520,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":4689,"internal_metadata":27651503},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:43:59.960 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:00.248 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:44:00.248 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:44:00.265 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T04:43:59.043919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":557312,"kb_used_data":29116,"kb_used_omap":27,"kb_used_meta":527780,"kb_avail":1509392128,"statfs":{"total":1546188226560,"available":1545617539072,"internally_reserved":0,"allocated":29814784,"data_stored":23894338,"data_compressed":6522539,"data_compressed_allocated":8884224,"data_compressed_original":18628608,"omap_allocated":28617,"internal_metadata":540446775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001993"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.971837+0000","last_change":"2024-09-16T04:43:23.308987+0000","last_active":"2024-09-16T04:43:55.971837+0000","last_peered":"2024-09-16T04:43:55.971837+0000","last_clean":"2024-09-16T04:43:55.971837+0000","last_became_active":"2024-09-16T04:43:23.208392+0000","last_became_peered":"2024-09-16T04:43:23.208392+0000","last_unstale":"2024-09-16T04:43:55.971837+0000","last_undegraded":"2024-09-16T04:43:55.971837+0000","last_fullsized":"2024-09-16T04:43:55.971837+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:07.865595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":281,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.654448+0000","last_change":"2024-09-16T04:43:13.855573+0000","last_active":"2024-09-16T04:43:55.654448+0000","last_peered":"2024-09-16T04:43:55.654448+0000","last_clean":"2024-09-16T04:43:55.654448+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:43:55.654448+0000","last_undegraded":"2024-09-16T04:43:55.654448+0000","last_fullsized":"2024-09-16T04:43:55.654448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.855454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:18.040654+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":283,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555167+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:43:55.555167+0000","last_peered":"2024-09-16T04:43:55.555167+0000","last_clean":"2024-09-16T04:43:55.555167+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:43:55.555167+0000","last_undegraded":"2024-09-16T04:43:55.555167+0000","last_fullsized":"2024-09-16T04:43:55.555167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":281,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.452532+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:43:54.452532+0000","last_peered":"2024-09-16T04:43:54.452532+0000","last_clean":"2024-09-16T04:43:54.452532+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:43:54.452532+0000","last_undegraded":"2024-09-16T04:43:54.452532+0000","last_fullsized":"2024-09-16T04:43:54.452532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":291,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585320+0000","last_change":"2024-09-16T04:43:50.333177+0000","last_active":"2024-09-16T04:43:54.585320+0000","last_peered":"2024-09-16T04:43:54.585320+0000","last_clean":"2024-09-16T04:43:54.585320+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:43:54.585320+0000","last_undegraded":"2024-09-16T04:43:54.585320+0000","last_fullsized":"2024-09-16T04:43:54.585320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:43:50.332925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:50.332925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:50.759622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":284,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.656155+0000","last_change":"2024-09-16T04:43:13.999043+0000","last_active":"2024-09-16T04:43:55.656155+0000","last_peered":"2024-09-16T04:43:55.656155+0000","last_clean":"2024-09-16T04:43:55.656155+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:43:55.656155+0000","last_undegraded":"2024-09-16T04:43:55.656155+0000","last_fullsized":"2024-09-16T04:43:55.656155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.998660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.998660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.784972+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":285,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585283+0000","last_change":"2024-09-16T04:43:22.549838+0000","last_active":"2024-09-16T04:43:54.585283+0000","last_peered":"2024-09-16T04:43:54.585283+0000","last_clean":"2024-09-16T04:43:54.585283+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:43:54.585283+0000","last_undegraded":"2024-09-16T04:43:54.585283+0000","last_fullsized":"2024-09-16T04:43:54.585283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:22.549596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:22.549596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:29.819458+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"111'2","reported_seq":304,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555236+0000","last_change":"2024-09-16T04:43:23.813052+0000","last_active":"2024-09-16T04:43:55.555236+0000","last_peered":"2024-09-16T04:43:55.555236+0000","last_clean":"2024-09-16T04:43:55.555236+0000","last_became_active":"2024-09-16T04:43:23.306721+0000","last_became_peered":"2024-09-16T04:43:23.306721+0000","last_unstale":"2024-09-16T04:43:55.555236+0000","last_undegraded":"2024-09-16T04:43:55.555236+0000","last_fullsized":"2024-09-16T04:43:55.555236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.979750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:24.154181+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":486,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.756267+0000","last_change":"2024-09-16T04:43:39.335810+0000","last_active":"2024-09-16T04:43:55.756267+0000","last_peered":"2024-09-16T04:43:55.756267+0000","last_clean":"2024-09-16T04:43:55.756267+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:43:55.756267+0000","last_undegraded":"2024-09-16T04:43:55.756267+0000","last_fullsized":"2024-09-16T04:43:55.756267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:43:39.335522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:43:39.335522+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:44.603174+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37164,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334676,"statfs":{"total":96636764160,"available":96598708224,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33016,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":119,"seq":511101108236,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28860,"kb_used_data":1820,"kb_used_omap":4,"kb_used_meta":27003,"kb_avail":94342980,"statfs":{"total":96636764160,"available":96607211520,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":4689,"internal_metadata":27651503},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:00.266 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:44:00.266 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:00.559 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:44:00.559 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:44:00.576 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T04:43:59.043919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":557312,"kb_used_data":29116,"kb_used_omap":27,"kb_used_meta":527780,"kb_avail":1509392128,"statfs":{"total":1546188226560,"available":1545617539072,"internally_reserved":0,"allocated":29814784,"data_stored":23894338,"data_compressed":6522539,"data_compressed_allocated":8884224,"data_compressed_original":18628608,"omap_allocated":28617,"internal_metadata":540446775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001993"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.971837+0000","last_change":"2024-09-16T04:43:23.308987+0000","last_active":"2024-09-16T04:43:55.971837+0000","last_peered":"2024-09-16T04:43:55.971837+0000","last_clean":"2024-09-16T04:43:55.971837+0000","last_became_active":"2024-09-16T04:43:23.208392+0000","last_became_peered":"2024-09-16T04:43:23.208392+0000","last_unstale":"2024-09-16T04:43:55.971837+0000","last_undegraded":"2024-09-16T04:43:55.971837+0000","last_fullsized":"2024-09-16T04:43:55.971837+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:07.865595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":281,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.654448+0000","last_change":"2024-09-16T04:43:13.855573+0000","last_active":"2024-09-16T04:43:55.654448+0000","last_peered":"2024-09-16T04:43:55.654448+0000","last_clean":"2024-09-16T04:43:55.654448+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:43:55.654448+0000","last_undegraded":"2024-09-16T04:43:55.654448+0000","last_fullsized":"2024-09-16T04:43:55.654448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:43:13.855454+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.855454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:18.040654+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":283,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555167+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:43:55.555167+0000","last_peered":"2024-09-16T04:43:55.555167+0000","last_clean":"2024-09-16T04:43:55.555167+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:43:55.555167+0000","last_undegraded":"2024-09-16T04:43:55.555167+0000","last_fullsized":"2024-09-16T04:43:55.555167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":281,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.452532+0000","last_change":"2024-09-16T04:42:35.851564+0000","last_active":"2024-09-16T04:43:54.452532+0000","last_peered":"2024-09-16T04:43:54.452532+0000","last_clean":"2024-09-16T04:43:54.452532+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:43:54.452532+0000","last_undegraded":"2024-09-16T04:43:54.452532+0000","last_fullsized":"2024-09-16T04:43:54.452532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:35.851448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:35.851448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:01.412894+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":291,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585320+0000","last_change":"2024-09-16T04:43:50.333177+0000","last_active":"2024-09-16T04:43:54.585320+0000","last_peered":"2024-09-16T04:43:54.585320+0000","last_clean":"2024-09-16T04:43:54.585320+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:43:54.585320+0000","last_undegraded":"2024-09-16T04:43:54.585320+0000","last_fullsized":"2024-09-16T04:43:54.585320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:43:50.332925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:50.332925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:50.759622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":284,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.656155+0000","last_change":"2024-09-16T04:43:13.999043+0000","last_active":"2024-09-16T04:43:55.656155+0000","last_peered":"2024-09-16T04:43:55.656155+0000","last_clean":"2024-09-16T04:43:55.656155+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:43:55.656155+0000","last_undegraded":"2024-09-16T04:43:55.656155+0000","last_fullsized":"2024-09-16T04:43:55.656155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.998660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.998660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:17.784972+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":285,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-16T04:43:54.585283+0000","last_change":"2024-09-16T04:43:22.549838+0000","last_active":"2024-09-16T04:43:54.585283+0000","last_peered":"2024-09-16T04:43:54.585283+0000","last_clean":"2024-09-16T04:43:54.585283+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:43:54.585283+0000","last_undegraded":"2024-09-16T04:43:54.585283+0000","last_fullsized":"2024-09-16T04:43:54.585283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:22.549596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:22.549596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:29.819458+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"111'2","reported_seq":304,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.555236+0000","last_change":"2024-09-16T04:43:23.813052+0000","last_active":"2024-09-16T04:43:55.555236+0000","last_peered":"2024-09-16T04:43:55.555236+0000","last_clean":"2024-09-16T04:43:55.555236+0000","last_became_active":"2024-09-16T04:43:23.306721+0000","last_became_peered":"2024-09-16T04:43:23.306721+0000","last_unstale":"2024-09-16T04:43:55.555236+0000","last_undegraded":"2024-09-16T04:43:55.555236+0000","last_fullsized":"2024-09-16T04:43:55.555236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.979750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:24.154181+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":486,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T04:43:55.756267+0000","last_change":"2024-09-16T04:43:39.335810+0000","last_active":"2024-09-16T04:43:55.756267+0000","last_peered":"2024-09-16T04:43:55.756267+0000","last_clean":"2024-09-16T04:43:55.756267+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:43:55.756267+0000","last_undegraded":"2024-09-16T04:43:55.756267+0000","last_fullsized":"2024-09-16T04:43:55.756267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:43:39.335522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:43:39.335522+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:44.603174+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37164,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334676,"statfs":{"total":96636764160,"available":96598708224,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33016,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1912832,"data_stored":1953172,"data_compressed":434669,"data_compressed_allocated":589824,"data_compressed_original":1609728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":119,"seq":511101108236,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28860,"kb_used_data":1820,"kb_used_omap":4,"kb_used_meta":27003,"kb_avail":94342980,"statfs":{"total":96636764160,"available":96607211520,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":4689,"internal_metadata":27651503},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1863680,"data_stored":1433952,"data_compressed":405737,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33000,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1855488,"data_stored":1424263,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":1424244,"data_compressed":402724,"data_compressed_allocated":548864,"data_compressed_original":1097728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:00.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:44:00.873 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:44:00.874 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":157,"fsid":"bdbec34e-c8aa-4776-9d32-ea336a99655f","created":"2024-09-16T04:41:10.374765+0000","modified":"2024-09-16T04:43:55.433336+0000","last_up_change":"2024-09-16T04:43:12.718474+0000","last_in_change":"2024-09-16T04:43:22.084893+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:41:23.074365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:41:25.522114+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9529dc0-e1eb-4b2f-8297-45254ddbffa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6801","nonce":1228226348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6803","nonce":1228226348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6807","nonce":1228226348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1228226348},{"type":"v1","addr":"172.21.15.59:6805","nonce":1228226348}]},"public_addr":"172.21.15.59:6801/1228226348","cluster_addr":"172.21.15.59:6803/1228226348","heartbeat_back_addr":"172.21.15.59:6807/1228226348","heartbeat_front_addr":"172.21.15.59:6805/1228226348","state":["exists","up"]},{"osd":1,"uuid":"dceca258-f5a1-4e8d-be6e-9d38bc5f14e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6817","nonce":2731115256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6819","nonce":2731115256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6823","nonce":2731115256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":2731115256},{"type":"v1","addr":"172.21.15.139:6821","nonce":2731115256}]},"public_addr":"172.21.15.139:6817/2731115256","cluster_addr":"172.21.15.139:6819/2731115256","heartbeat_back_addr":"172.21.15.139:6823/2731115256","heartbeat_front_addr":"172.21.15.139:6821/2731115256","state":["exists","up"]},{"osd":2,"uuid":"6bfc934a-0ca2-4b1f-9e09-14c18a7d5116","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6825","nonce":76637888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6827","nonce":76637888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6831","nonce":76637888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":76637888},{"type":"v1","addr":"172.21.15.144:6829","nonce":76637888}]},"public_addr":"172.21.15.144:6825/76637888","cluster_addr":"172.21.15.144:6827/76637888","heartbeat_back_addr":"172.21.15.144:6831/76637888","heartbeat_front_addr":"172.21.15.144:6829/76637888","state":["exists","up"]},{"osd":3,"uuid":"0b949ef9-82e3-483f-8be6-65e04d169c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6803","nonce":58455506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6805","nonce":58455506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6809","nonce":58455506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":58455506},{"type":"v1","addr":"172.21.15.165:6807","nonce":58455506}]},"public_addr":"172.21.15.165:6803/58455506","cluster_addr":"172.21.15.165:6805/58455506","heartbeat_back_addr":"172.21.15.165:6809/58455506","heartbeat_front_addr":"172.21.15.165:6807/58455506","state":["exists","up"]},{"osd":4,"uuid":"d35e5791-7bb4-493d-99ec-3ff942117371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6825","nonce":2712399685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6827","nonce":2712399685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6831","nonce":2712399685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":2712399685},{"type":"v1","addr":"172.21.15.59:6829","nonce":2712399685}]},"public_addr":"172.21.15.59:6825/2712399685","cluster_addr":"172.21.15.59:6827/2712399685","heartbeat_back_addr":"172.21.15.59:6831/2712399685","heartbeat_front_addr":"172.21.15.59:6829/2712399685","state":["exists","up"]},{"osd":5,"uuid":"159725f2-3713-4f0e-9e44-55b678bf09fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6801","nonce":2946170755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6803","nonce":2946170755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6807","nonce":2946170755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2946170755},{"type":"v1","addr":"172.21.15.139:6805","nonce":2946170755}]},"public_addr":"172.21.15.139:6801/2946170755","cluster_addr":"172.21.15.139:6803/2946170755","heartbeat_back_addr":"172.21.15.139:6807/2946170755","heartbeat_front_addr":"172.21.15.139:6805/2946170755","state":["exists","up"]},{"osd":6,"uuid":"e642cfb3-cf94-4124-93db-9e2bdf1b7c5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6801","nonce":3812816248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6803","nonce":3812816248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6807","nonce":3812816248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":3812816248},{"type":"v1","addr":"172.21.15.144:6805","nonce":3812816248}]},"public_addr":"172.21.15.144:6801/3812816248","cluster_addr":"172.21.15.144:6803/3812816248","heartbeat_back_addr":"172.21.15.144:6807/3812816248","heartbeat_front_addr":"172.21.15.144:6805/3812816248","state":["exists","up"]},{"osd":7,"uuid":"8aabebf6-a2ea-44af-be5d-d7aaf7cdfecb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6819","nonce":1352862134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6821","nonce":1352862134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6825","nonce":1352862134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1352862134},{"type":"v1","addr":"172.21.15.165:6823","nonce":1352862134}]},"public_addr":"172.21.15.165:6819/1352862134","cluster_addr":"172.21.15.165:6821/1352862134","heartbeat_back_addr":"172.21.15.165:6825/1352862134","heartbeat_front_addr":"172.21.15.165:6823/1352862134","state":["exists","up"]},{"osd":8,"uuid":"04681c69-387b-4dd8-9eb4-687f34ba9dab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6817","nonce":1451673997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6819","nonce":1451673997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6823","nonce":1451673997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1451673997},{"type":"v1","addr":"172.21.15.59:6821","nonce":1451673997}]},"public_addr":"172.21.15.59:6817/1451673997","cluster_addr":"172.21.15.59:6819/1451673997","heartbeat_back_addr":"172.21.15.59:6823/1451673997","heartbeat_front_addr":"172.21.15.59:6821/1451673997","state":["exists","up"]},{"osd":9,"uuid":"713a3b76-8daf-4e44-9d0b-68369a9da692","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6809","nonce":174106571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6811","nonce":174106571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6815","nonce":174106571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":174106571},{"type":"v1","addr":"172.21.15.139:6813","nonce":174106571}]},"public_addr":"172.21.15.139:6809/174106571","cluster_addr":"172.21.15.139:6811/174106571","heartbeat_back_addr":"172.21.15.139:6815/174106571","heartbeat_front_addr":"172.21.15.139:6813/174106571","state":["exists","up"]},{"osd":10,"uuid":"81450492-3178-4643-8066-847fcdd266b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6809","nonce":66779937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6812","nonce":66779937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6818","nonce":66779937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":66779937},{"type":"v1","addr":"172.21.15.144:6815","nonce":66779937}]},"public_addr":"172.21.15.144:6809/66779937","cluster_addr":"172.21.15.144:6812/66779937","heartbeat_back_addr":"172.21.15.144:6818/66779937","heartbeat_front_addr":"172.21.15.144:6815/66779937","state":["exists","up"]},{"osd":11,"uuid":"9938b272-43e2-4b48-91ca-42cbe5f26664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6811","nonce":49204916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6813","nonce":49204916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6817","nonce":49204916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":49204916},{"type":"v1","addr":"172.21.15.165:6815","nonce":49204916}]},"public_addr":"172.21.15.165:6811/49204916","cluster_addr":"172.21.15.165:6813/49204916","heartbeat_back_addr":"172.21.15.165:6817/49204916","heartbeat_front_addr":"172.21.15.165:6815/49204916","state":["exists","up"]},{"osd":12,"uuid":"cd20410e-c17f-43d7-8512-9efd8193a251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6809","nonce":3097346146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6811","nonce":3097346146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6815","nonce":3097346146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3097346146},{"type":"v1","addr":"172.21.15.59:6813","nonce":3097346146}]},"public_addr":"172.21.15.59:6809/3097346146","cluster_addr":"172.21.15.59:6811/3097346146","heartbeat_back_addr":"172.21.15.59:6815/3097346146","heartbeat_front_addr":"172.21.15.59:6813/3097346146","state":["exists","up"]},{"osd":13,"uuid":"953622d2-9e72-451d-8863-d8c55139c65b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6825","nonce":2903817823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6827","nonce":2903817823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6831","nonce":2903817823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2903817823},{"type":"v1","addr":"172.21.15.139:6829","nonce":2903817823}]},"public_addr":"172.21.15.139:6825/2903817823","cluster_addr":"172.21.15.139:6827/2903817823","heartbeat_back_addr":"172.21.15.139:6831/2903817823","heartbeat_front_addr":"172.21.15.139:6829/2903817823","state":["exists","up"]},{"osd":14,"uuid":"e6388924-0444-47c0-b61c-7ca66661237f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6811","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6814","nonce":229023981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6819","nonce":229023981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6823","nonce":229023981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":229023981},{"type":"v1","addr":"172.21.15.144:6821","nonce":229023981}]},"public_addr":"172.21.15.144:6814/229023981","cluster_addr":"172.21.15.144:6819/229023981","heartbeat_back_addr":"172.21.15.144:6823/229023981","heartbeat_front_addr":"172.21.15.144:6821/229023981","state":["exists","up"]},{"osd":15,"uuid":"8335941d-9563-4527-90f6-3b7f9129f530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":99,"up_from":119,"up_thru":124,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6827","nonce":2616552093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6829","nonce":2616552093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6833","nonce":2616552093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":2616552093},{"type":"v1","addr":"172.21.15.165:6831","nonce":2616552093}]},"public_addr":"172.21.15.165:6827/2616552093","cluster_addr":"172.21.15.165:6829/2616552093","heartbeat_back_addr":"172.21.15.165:6833/2616552093","heartbeat_front_addr":"172.21.15.165:6831/2616552093","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:48.826878+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:40.740589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:44.548804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:53.306808+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:42.924457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:47.648804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:49.832912+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:48.874174+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:38.827784+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:55.426432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:37.768155+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:41.884656+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:48.659730+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:51.158730+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:42:52.799239+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:43:12.392064+0000","dead_epoch":101}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3,"length":1},{"begin":7,"length":1},{"begin":9,"length":16},{"begin":26,"length":1},{"begin":28,"length":1},{"begin":35,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:44:01.892 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:44:01.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:02.060 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:02.060 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:02.061 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:02.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:44:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:44:02.379 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T04:44:02.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:02.532 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:02.532 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:02.532 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:02.547 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T04:44:02.842 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T04:44:02.862 INFO:tasks.ceph:Scrubbing osd.8 2024-09-16T04:44:02.862 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:03.009 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:03.010 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:03.010 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:03.026 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-16T04:44:03.314 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 8 to deep-scrub 2024-09-16T04:44:03.333 INFO:tasks.ceph:Scrubbing osd.12 2024-09-16T04:44:03.333 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:03.488 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:03.488 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:03.488 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:03.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-16T04:44:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 12 to deep-scrub 2024-09-16T04:44:03.803 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:44:03.803 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:03.957 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:03.957 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:03.957 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:03.974 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:44:04.270 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:44:04.288 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T04:44:04.289 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:04.445 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:04.446 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:04.446 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:04.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T04:44:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T04:44:04.775 INFO:tasks.ceph:Scrubbing osd.9 2024-09-16T04:44:04.776 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:04.933 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:04.933 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:04.933 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:04.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-16T04:44:05.249 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 9 to deep-scrub 2024-09-16T04:44:05.269 INFO:tasks.ceph:Scrubbing osd.13 2024-09-16T04:44:05.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:05.427 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:05.427 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:05.427 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:05.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-16T04:44:05.746 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 13 to deep-scrub 2024-09-16T04:44:05.766 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:44:05.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:05.927 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:05.927 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:05.927 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:05.945 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:44:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:44:06.271 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T04:44:06.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:06.434 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:06.434 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:06.434 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:06.451 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T04:44:06.745 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T04:44:06.763 INFO:tasks.ceph:Scrubbing osd.10 2024-09-16T04:44:06.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:06.916 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:06.917 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:06.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-16T04:44:07.214 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 10 to deep-scrub 2024-09-16T04:44:07.231 INFO:tasks.ceph:Scrubbing osd.14 2024-09-16T04:44:07.231 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:07.386 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:07.386 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:07.386 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:07.401 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-16T04:44:07.698 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 14 to deep-scrub 2024-09-16T04:44:07.718 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:44:07.718 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:07.878 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:07.878 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:07.879 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:07.895 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:44:08.198 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:44:08.218 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T04:44:08.218 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:08.377 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:08.377 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:08.378 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:08.394 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T04:44:08.692 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T04:44:08.712 INFO:tasks.ceph:Scrubbing osd.11 2024-09-16T04:44:08.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:08.863 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:08.863 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:08.863 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:08.880 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-16T04:44:09.184 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 11 to deep-scrub 2024-09-16T04:44:09.203 INFO:tasks.ceph:Scrubbing osd.15 2024-09-16T04:44:09.203 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:44:09.360 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-16T04:44:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:44:09.361 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-16T04:44:09.378 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-16T04:44:09.685 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 15 to deep-scrub 2024-09-16T04:44:09.705 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:44:09.993 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:44:10.009 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-16T04:44:09.046935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":557564,"kb_used_data":29440,"kb_used_omap":27,"kb_used_meta":527780,"kb_avail":1509391876,"statfs":{"total":1546188226560,"available":1545617281024,"internally_reserved":0,"allocated":30146560,"data_stored":24156454,"data_compressed":6603874,"data_compressed_allocated":8994816,"data_compressed_original":18849792,"omap_allocated":28617,"internal_metadata":540446775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003016"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":73,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:03.208704+0000","last_change":"2024-09-16T04:43:23.308987+0000","last_active":"2024-09-16T04:44:03.208704+0000","last_peered":"2024-09-16T04:44:03.208704+0000","last_clean":"2024-09-16T04:44:03.208704+0000","last_became_active":"2024-09-16T04:43:23.208392+0000","last_became_peered":"2024-09-16T04:43:23.208392+0000","last_unstale":"2024-09-16T04:44:03.208704+0000","last_undegraded":"2024-09-16T04:44:03.208704+0000","last_fullsized":"2024-09-16T04:44:03.208704+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.667111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.667111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:07.865595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":292,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:03.885068+0000","last_change":"2024-09-16T04:44:03.885127+0000","last_active":"2024-09-16T04:44:03.885068+0000","last_peered":"2024-09-16T04:44:03.885068+0000","last_clean":"2024-09-16T04:44:03.885068+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:44:03.885068+0000","last_undegraded":"2024-09-16T04:44:03.885068+0000","last_fullsized":"2024-09-16T04:44:03.885068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:03.884846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:03.884846+0000","last_clean_scrub_stamp":"2024-09-16T04:44:03.884846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:09.043114+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":285,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:02.556249+0000","last_change":"2024-09-16T04:42:47.543520+0000","last_active":"2024-09-16T04:44:02.556249+0000","last_peered":"2024-09-16T04:44:02.556249+0000","last_clean":"2024-09-16T04:44:02.556249+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:44:02.556249+0000","last_undegraded":"2024-09-16T04:44:02.556249+0000","last_fullsized":"2024-09-16T04:44:02.556249+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:42:47.543408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:42:47.543408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:09.977776+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":294,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:03.673425+0000","last_change":"2024-09-16T04:44:03.673497+0000","last_active":"2024-09-16T04:44:03.673425+0000","last_peered":"2024-09-16T04:44:03.673425+0000","last_clean":"2024-09-16T04:44:03.673425+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:44:03.673425+0000","last_undegraded":"2024-09-16T04:44:03.673425+0000","last_fullsized":"2024-09-16T04:44:03.673425+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:03.673255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:03.673255+0000","last_clean_scrub_stamp":"2024-09-16T04:44:03.673255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:13.698757+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":295,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:02.672062+0000","last_change":"2024-09-16T04:43:50.333177+0000","last_active":"2024-09-16T04:44:02.672062+0000","last_peered":"2024-09-16T04:44:02.672062+0000","last_clean":"2024-09-16T04:44:02.672062+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:44:02.672062+0000","last_undegraded":"2024-09-16T04:44:02.672062+0000","last_fullsized":"2024-09-16T04:44:02.672062+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T04:43:50.332925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:50.332925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:50.759622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":287,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:02.481893+0000","last_change":"2024-09-16T04:43:13.999043+0000","last_active":"2024-09-16T04:44:02.481893+0000","last_peered":"2024-09-16T04:44:02.481893+0000","last_clean":"2024-09-16T04:44:02.481893+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:44:02.481893+0000","last_undegraded":"2024-09-16T04:44:02.481893+0000","last_fullsized":"2024-09-16T04:44:02.481893+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:13.998660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:13.998660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:02.672062+0000","last_change":"2024-09-16T04:43:22.549838+0000","last_active":"2024-09-16T04:44:02.672062+0000","last_peered":"2024-09-16T04:44:02.672062+0000","last_clean":"2024-09-16T04:44:02.672062+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:44:02.672062+0000","last_undegraded":"2024-09-16T04:44:02.672062+0000","last_fullsized":"2024-09-16T04:44:02.672062+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:43:22.549596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:26.370698+0000","last_clean_scrub_stamp":"2024-09-16T04:43:22.549596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:44:29.819458+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"111'2","reported_seq":307,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:02.657123+0000","last_change":"2024-09-16T04:43:23.813052+0000","last_active":"2024-09-16T04:44:02.657123+0000","last_peered":"2024-09-16T04:44:02.657123+0000","last_clean":"2024-09-16T04:44:02.657123+0000","last_became_active":"2024-09-16T04:43:23.306721+0000","last_became_peered":"2024-09-16T04:43:23.306721+0000","last_unstale":"2024-09-16T04:44:02.657123+0000","last_undegraded":"2024-09-16T04:44:02.657123+0000","last_fullsized":"2024-09-16T04:44:02.657123+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:43:01.979750+0000","last_clean_scrub_stamp":"2024-09-16T04:43:01.979750+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"32'157","reported_seq":489,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-16T04:44:02.481976+0000","last_change":"2024-09-16T04:43:39.335810+0000","last_active":"2024-09-16T04:44:02.481976+0000","last_peered":"2024-09-16T04:44:02.481976+0000","last_clean":"2024-09-16T04:44:02.481976+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:44:02.481976+0000","last_undegraded":"2024-09-16T04:44:02.481976+0000","last_fullsized":"2024-09-16T04:44:02.481976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'157","last_scrub_stamp":"2024-09-16T04:43:39.335522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:41:23.354934+0000","last_clean_scrub_stamp":"2024-09-16T04:43:39.335522+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1937408,"data_stored":1972588,"data_compressed":440694,"data_compressed_allocated":598016,"data_compressed_original":1626112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247846,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37180,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334660,"statfs":{"total":96636764160,"available":96598691840,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247846,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37180,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334660,"statfs":{"total":96636764160,"available":96598691840,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37180,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334660,"statfs":{"total":96636764160,"available":96598691840,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313253,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33020,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1937408,"data_stored":1972588,"data_compressed":440694,"data_compressed_allocated":598016,"data_compressed_original":1626112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":119,"seq":511101108238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28860,"kb_used_data":1832,"kb_used_omap":4,"kb_used_meta":27003,"kb_avail":94342980,"statfs":{"total":96636764160,"available":96607211520,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":4689,"internal_metadata":27651503},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1443679,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37180,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334660,"statfs":{"total":96636764160,"available":96598691840,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33020,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280548,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33020,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280549,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37180,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334660,"statfs":{"total":96636764160,"available":96598691840,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37176,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334664,"statfs":{"total":96636764160,"available":96598695936,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280549,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33020,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1443679,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280549,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37180,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334660,"statfs":{"total":96636764160,"available":96598691840,"internally_reserved":0,"allocated":1875968,"data_stored":1443660,"data_compressed":408749,"data_compressed_allocated":557056,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:10.013 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T04:42:47.667111+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=42, tm_sec=47, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=44, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:44:10.014 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T04:42:47.543408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=42, tm_sec=47, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=44, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:44:10.014 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T04:43:50.332925+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=43, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=44, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:44:10.014 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T04:43:13.998660+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=43, tm_sec=13, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=44, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:44:10.014 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T04:43:22.549596+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=43, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=44, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:44:10.014 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T04:43:01.979750+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=43, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=44, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:44:10.014 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T04:43:39.335522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=43, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=44, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:44:10.014 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:44:30.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:44:30.327 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-16T04:44:30.327 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-16T04:44:30.346 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T04:44:29.052962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":205,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":558336,"kb_used_data":30208,"kb_used_omap":27,"kb_used_meta":527780,"kb_avail":1509391104,"statfs":{"total":1546188226560,"available":1545616490496,"internally_reserved":0,"allocated":30932992,"data_stored":24777766,"data_compressed":6796578,"data_compressed_allocated":9256960,"data_compressed_original":19374080,"omap_allocated":28617,"internal_metadata":540446775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002912"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":98,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.633692+0000","last_change":"2024-09-16T04:44:11.367298+0000","last_active":"2024-09-16T04:44:20.633692+0000","last_peered":"2024-09-16T04:44:20.633692+0000","last_clean":"2024-09-16T04:44:20.633692+0000","last_became_active":"2024-09-16T04:43:23.208392+0000","last_became_peered":"2024-09-16T04:43:23.208392+0000","last_unstale":"2024-09-16T04:44:20.633692+0000","last_undegraded":"2024-09-16T04:44:20.633692+0000","last_fullsized":"2024-09-16T04:44:20.633692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:11.367063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:11.367063+0000","last_clean_scrub_stamp":"2024-09-16T04:44:11.367063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:27.839060+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":300,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.732274+0000","last_change":"2024-09-16T04:44:03.885127+0000","last_active":"2024-09-16T04:44:20.732274+0000","last_peered":"2024-09-16T04:44:20.732274+0000","last_clean":"2024-09-16T04:44:20.732274+0000","last_became_active":"2024-09-16T04:41:27.599729+0000","last_became_peered":"2024-09-16T04:41:27.599729+0000","last_unstale":"2024-09-16T04:44:20.732274+0000","last_undegraded":"2024-09-16T04:44:20.732274+0000","last_fullsized":"2024-09-16T04:44:20.732274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:03.884846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:03.884846+0000","last_clean_scrub_stamp":"2024-09-16T04:44:03.884846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:09.043114+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0.00049610300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":302,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.632096+0000","last_change":"2024-09-16T04:44:07.016364+0000","last_active":"2024-09-16T04:44:20.632096+0000","last_peered":"2024-09-16T04:44:20.632096+0000","last_clean":"2024-09-16T04:44:20.632096+0000","last_became_active":"2024-09-16T04:41:27.601148+0000","last_became_peered":"2024-09-16T04:41:27.601148+0000","last_unstale":"2024-09-16T04:44:20.632096+0000","last_undegraded":"2024-09-16T04:44:20.632096+0000","last_fullsized":"2024-09-16T04:44:20.632096+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:07.016037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:07.016037+0000","last_clean_scrub_stamp":"2024-09-16T04:44:07.016037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:35.978533+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.10101020099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":302,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.217370+0000","last_change":"2024-09-16T04:44:03.673497+0000","last_active":"2024-09-16T04:44:20.217370+0000","last_peered":"2024-09-16T04:44:20.217370+0000","last_clean":"2024-09-16T04:44:20.217370+0000","last_became_active":"2024-09-16T04:41:27.820647+0000","last_became_peered":"2024-09-16T04:41:27.820647+0000","last_unstale":"2024-09-16T04:44:20.217370+0000","last_undegraded":"2024-09-16T04:44:20.217370+0000","last_fullsized":"2024-09-16T04:44:20.217370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:03.673255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:03.673255+0000","last_clean_scrub_stamp":"2024-09-16T04:44:03.673255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:13.698757+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.00039964599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"158'3","reported_seq":314,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.881427+0000","last_change":"2024-09-16T04:44:10.401886+0000","last_active":"2024-09-16T04:44:20.881427+0000","last_peered":"2024-09-16T04:44:20.881427+0000","last_clean":"2024-09-16T04:44:20.881427+0000","last_became_active":"2024-09-16T04:41:27.813329+0000","last_became_peered":"2024-09-16T04:41:27.813329+0000","last_unstale":"2024-09-16T04:44:20.881427+0000","last_undegraded":"2024-09-16T04:44:20.881427+0000","last_fullsized":"2024-09-16T04:44:20.881427+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2024-09-16T04:44:10.401636+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2024-09-16T04:44:10.401636+0000","last_clean_scrub_stamp":"2024-09-16T04:44:10.401636+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:12.522023+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0015670219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":303,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.633840+0000","last_change":"2024-09-16T04:44:07.510902+0000","last_active":"2024-09-16T04:44:20.633840+0000","last_peered":"2024-09-16T04:44:20.633840+0000","last_clean":"2024-09-16T04:44:20.633840+0000","last_became_active":"2024-09-16T04:41:27.500507+0000","last_became_peered":"2024-09-16T04:41:27.500507+0000","last_unstale":"2024-09-16T04:44:20.633840+0000","last_undegraded":"2024-09-16T04:44:20.633840+0000","last_fullsized":"2024-09-16T04:44:20.633840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:07.510637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:07.510637+0000","last_clean_scrub_stamp":"2024-09-16T04:44:07.510637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:30.805304+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0.00096766099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":306,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.881409+0000","last_change":"2024-09-16T04:44:09.113533+0000","last_active":"2024-09-16T04:44:20.881409+0000","last_peered":"2024-09-16T04:44:20.881409+0000","last_clean":"2024-09-16T04:44:20.881409+0000","last_became_active":"2024-09-16T04:41:27.500359+0000","last_became_peered":"2024-09-16T04:41:27.500359+0000","last_unstale":"2024-09-16T04:44:20.881409+0000","last_undegraded":"2024-09-16T04:44:20.881409+0000","last_fullsized":"2024-09-16T04:44:20.881409+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:44:09.113195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:44:09.113195+0000","last_clean_scrub_stamp":"2024-09-16T04:44:09.113195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:31.528406+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.101072997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"111'2","reported_seq":323,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.120699+0000","last_change":"2024-09-16T04:44:05.475260+0000","last_active":"2024-09-16T04:44:20.120699+0000","last_peered":"2024-09-16T04:44:20.120699+0000","last_clean":"2024-09-16T04:44:20.120699+0000","last_became_active":"2024-09-16T04:43:23.306721+0000","last_became_peered":"2024-09-16T04:43:23.306721+0000","last_unstale":"2024-09-16T04:44:20.120699+0000","last_undegraded":"2024-09-16T04:44:20.120699+0000","last_fullsized":"2024-09-16T04:44:20.120699+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-09-16T04:44:05.474973+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-09-16T04:44:05.474973+0000","last_clean_scrub_stamp":"2024-09-16T04:44:05.474973+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:32.812096+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.001002632,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"158'159","reported_seq":507,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T04:44:20.633860+0000","last_change":"2024-09-16T04:44:06.596060+0000","last_active":"2024-09-16T04:44:20.633860+0000","last_peered":"2024-09-16T04:44:20.633860+0000","last_clean":"2024-09-16T04:44:20.633860+0000","last_became_active":"2024-09-16T04:41:26.343313+0000","last_became_peered":"2024-09-16T04:41:26.343313+0000","last_unstale":"2024-09-16T04:44:20.633860+0000","last_undegraded":"2024-09-16T04:44:20.633860+0000","last_fullsized":"2024-09-16T04:44:20.633860+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'159","last_scrub_stamp":"2024-09-16T04:44:06.595707+0000","last_deep_scrub":"158'159","last_deep_scrub_stamp":"2024-09-16T04:44:06.595707+0000","last_clean_scrub_stamp":"2024-09-16T04:44:06.595707+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:45:28.444605+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":63890,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":23,"seq":98784247850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37288,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334552,"statfs":{"total":96636764160,"available":96598581248,"internally_reserved":0,"allocated":1986560,"data_stored":2011420,"data_compressed":452738,"data_compressed_allocated":614400,"data_compressed_original":1658880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":23,"seq":98784247850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313257,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33068,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[4,5,6,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1986560,"data_stored":2011420,"data_compressed":452738,"data_compressed_allocated":614400,"data_compressed_original":1658880,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":119,"seq":511101108242,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28908,"kb_used_data":1880,"kb_used_omap":4,"kb_used_meta":27003,"kb_avail":94342932,"statfs":{"total":96636764160,"available":96607162368,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":4689,"internal_metadata":27651503},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1929216,"data_stored":1482511,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,6,7,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280553,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280553,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33068,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1929216,"data_stored":1482511,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280553,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1482492,"data_compressed":420793,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:44:30.348 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:44:30.682 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:44:30.683 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:44:30.683 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:44:30.683 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:30.759 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:44:30.759 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T04:44:30.759 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:30.858 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T04:44:30.858 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-16T04:44:30.858 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:36.960 INFO:tasks.ceph.osd.8:Stopped 2024-09-16T04:44:36.960 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-16T04:44:36.960 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:37.045 INFO:tasks.ceph.osd.12:Stopped 2024-09-16T04:44:37.045 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:44:37.045 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:37.117 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:44:37.117 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T04:44:37.117 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:37.214 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T04:44:37.214 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-16T04:44:37.214 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:37.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2024-09-16T04:44:43.316 INFO:tasks.ceph.osd.9:Stopped 2024-09-16T04:44:43.316 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-16T04:44:43.316 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:43.400 INFO:tasks.ceph.osd.13:Stopped 2024-09-16T04:44:43.400 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:44:43.401 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:43.490 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:44:43.491 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T04:44:43.491 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:43.566 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T04:44:43.566 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-16T04:44:43.566 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:43.664 INFO:tasks.ceph.osd.10:Stopped 2024-09-16T04:44:43.664 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-16T04:44:43.664 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:43.760 INFO:tasks.ceph.osd.14:Stopped 2024-09-16T04:44:43.760 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:44:43.760 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:43.843 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:44:43.843 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T04:44:43.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:43.925 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T04:44:43.926 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-16T04:44:43.926 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:44.008 INFO:tasks.ceph.osd.11:Stopped 2024-09-16T04:44:44.008 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-16T04:44:44.008 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:44.102 INFO:tasks.ceph.osd.15:Stopped 2024-09-16T04:44:44.103 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:44:44.103 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T04:44:44.103 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:44.132 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T04:44:44.132 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:44:44.132 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:44.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2024-09-16T04:44:44.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-09-16T04:44:44.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-09-16T04:44:44.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 has been restored 2024-09-16T04:44:44.165 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:44:44.165 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:44:44.165 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:44:44.165 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:44.188 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:44:44.189 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:44:44.189 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:44.215 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:44:44.215 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:44:44.215 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:44:44.237 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:44:44.237 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:44:44.238 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-16T04:44:44.257 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:44:44.257 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:44:44.607 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:44:44.607 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T04:44:44.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:44:44.797 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-16T04:44:45.063 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:44:45.063 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-16T04:44:45.249 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:44:45.249 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:44:45.430 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:44:45.430 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T04:44:45.596 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:44:45.596 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-16T04:44:45.730 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:44:45.731 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-16T04:44:45.889 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:44:45.889 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:44:46.084 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:44:46.084 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T04:44:46.273 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:44:46.274 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-16T04:44:46.473 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:44:46.473 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-16T04:44:46.677 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:44:46.677 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:44:46.893 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:44:46.893 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T04:44:47.093 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:44:47.093 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-16T04:44:47.266 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:44:47.266 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-16T04:44:47.440 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:44:47.441 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/data/mon.a.tgz 2024-09-16T04:44:47.441 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-09-16T04:44:47.447 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.4heIEbudbH 2024-09-16T04:44:47.448 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.4heIEbudbH 2024-09-16T04:44:47.620 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.4heIEbudbH 2024-09-16T04:44:47.683 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.4heIEbudbH is 507KB 2024-09-16T04:44:47.744 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.4heIEbudbH 2024-09-16T04:44:47.749 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/data/mon.b.tgz 2024-09-16T04:44:47.749 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-09-16T04:44:47.755 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.Aby3bs1aUG 2024-09-16T04:44:47.755 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Aby3bs1aUG 2024-09-16T04:44:47.968 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.Aby3bs1aUG 2024-09-16T04:44:48.034 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.Aby3bs1aUG is 695KB 2024-09-16T04:44:48.100 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.Aby3bs1aUG 2024-09-16T04:44:48.105 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/data/mon.c.tgz 2024-09-16T04:44:48.106 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-09-16T04:44:48.111 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.Z1kQDhEEw5 2024-09-16T04:44:48.111 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Z1kQDhEEw5 2024-09-16T04:44:48.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.Z1kQDhEEw5 2024-09-16T04:44:48.376 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.Z1kQDhEEw5 is 696KB 2024-09-16T04:44:48.439 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.Z1kQDhEEw5 2024-09-16T04:44:48.443 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:44:48.444 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:44:48.446 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:44:48.448 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:44:48.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:44:48.506 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:44:48.512 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:44:48.545 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:44:48.550 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:44:48.550 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:44:48.552 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi059/crash 2024-09-16T04:44:48.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:44:48.567 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi139/crash 2024-09-16T04:44:48.567 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:44:48.582 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi144/crash 2024-09-16T04:44:48.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:44:48.600 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi165/crash 2024-09-16T04:44:48.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:44:48.614 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:44:48.614 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:44:48.616 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:44:48.627 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:44:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36433.log 2024-09-16T04:44:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29992.log 2024-09-16T04:44:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35144.log 2024-09-16T04:44:48.639 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36433.log: /var/log/ceph/ceph-client.admin.29992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36433.log.gz 2024-09-16T04:44:48.639 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29992.log.gz 2024-09-16T04:44:48.639 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32037.log 2024-09-16T04:44:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-09-16T04:44:48.640 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35144.log.gz 2024-09-16T04:44:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36511.log 2024-09-16T04:44:48.640 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32037.log.gz 2024-09-16T04:44:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34099.log 2024-09-16T04:44:48.641 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36511.log.gz 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29902.log 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36225.log 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34099.log.gz 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37581.log 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29902.log.gz 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36225.log.gz 2024-09-16T04:44:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30958.log 2024-09-16T04:44:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-09-16T04:44:48.644 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37581.log.gz 2024-09-16T04:44:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T04:44:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30958.log.gz 2024-09-16T04:44:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-09-16T04:44:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33374.log 2024-09-16T04:44:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37511.log 2024-09-16T04:44:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38461.log 2024-09-16T04:44:48.646 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33374.log.gz 2024-09-16T04:44:48.646 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39085.log 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37511.log.gz 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37941.log 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38461.log.gz 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34150.log 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39085.log.gz 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37138.log 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37941.log.gz 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38981.log 0.0% 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34150.log.gz 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29999.log 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37138.log.gz 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38981.log.gz 2024-09-16T04:44:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30379.log 2024-09-16T04:44:48.650 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34495.log 2024-09-16T04:44:48.650 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29999.log.gz 2024-09-16T04:44:48.650 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34397.log 2024-09-16T04:44:48.650 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30379.log.gz 2024-09-16T04:44:48.650 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34495.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34495.log.gz 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32934.log 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34397.log.gz 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.0.log: 93.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33813.log 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32934.log.gz 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35981.log 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.33813.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31880.log 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33813.log.gz 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35981.log.gz 2024-09-16T04:44:48.651 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37154.log 2024-09-16T04:44:48.652 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:44:48.656 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T04:44:48.656 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:44:48.656 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:44:48.656 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:44:48.656 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-16T04:44:48.656 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-16T04:44:48.657 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:44:48.657 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:44:48.657 INFO:teuthology.orchestra.run.smithi139.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:44:48.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38747.log 2024-09-16T04:44:48.668 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31880.log.gz 2024-09-16T04:44:48.668 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34972.log 2024-09-16T04:44:48.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37154.log.gz 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30956.log 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38747.log.gz 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34972.log.gz 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37576.log 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30939.log 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30956.log.gz 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37576.log.gz 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37168.log 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30939.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.30939.log.gz /var/log/ceph/ceph-client.admin.39059.log 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37168.log.gz 2024-09-16T04:44:48.669 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35836.log 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39059.log.gz 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30796.log 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36641.log 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35836.log.gz 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34212.log 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30796.log.gz 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36641.log.gz 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30907.log 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33989.log 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34212.log.gz 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32360.log 2024-09-16T04:44:48.670 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30907.log.gz 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33989.log.gz 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29975.log 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38082.log 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32360.log.gz 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31814.log 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29975.log.gz 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38082.log.gz 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33922.log 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31814.log.gz 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36693.log 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39371.log 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33922.log.gz 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37142.log 2024-09-16T04:44:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36693.log.gz 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39371.log.gz 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35347.log 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31286.log 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37142.log.gz 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35347.log.gz 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31286.log.gz 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36381.log 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39241.log 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.8.log: /var/log/ceph/ceph-client.admin.36381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35321.log 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36381.log.gz 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32172.log 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39241.log.gz 2024-09-16T04:44:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35321.log.gz 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32233.log 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36953.log 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32172.log.gz 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32233.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32233.log.gz 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30367.log 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36953.log.gz 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37538.log 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34549.log 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30367.log.gz 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37538.log.gz 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33223.log 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37577.log 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34549.log.gz 2024-09-16T04:44:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33223.log.gz 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31717.log 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35082.log 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31717.log.gz 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38851.log 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37150.log 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35082.log.gz 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38851.log.gz 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38903.log 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37150.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37150.log.gz 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38903.log.gz 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36849.log 2024-09-16T04:44:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2024-09-16T04:44:48.675 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36849.log.gz 2024-09-16T04:44:48.675 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33483.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.14.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-16T04:44:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.10.log: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:44:48.681 INFO:teuthology.orchestra.run.smithi144.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:44:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30491.log 2024-09-16T04:44:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31979.log 2024-09-16T04:44:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26552.log 2024-09-16T04:44:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/objectstore_tool.30491.log: /var/log/ceph/objectstore_tool.31979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26256.log 2024-09-16T04:44:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26552.log.gz 2024-09-16T04:44:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T04:44:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30038.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-client.admin.26256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26256.log.gz 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26208.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.0.30038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26504.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.30038.log.gz 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26383.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26208.log.gz 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26504.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.33517.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26504.log.gz 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26383.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26383.log.gz 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/bluestore_tool.33517.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31235.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:gzip/var/log/ceph/ceph-osd.11.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26184.log 2024-09-16T04:44:48.695 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/objectstore_tool.31235.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.33890.log 2024-09-16T04:44:48.696 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26184.log.gz 2024-09-16T04:44:48.696 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26431.log 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/bluestore_tool.33890.log: 93.6% -- replaced with /var/log/ceph/bluestore_tool.33517.log.gz 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26431.log.gz 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.15.log: 93.5% -- replaced with /var/log/ceph/bluestore_tool.33890.log.gz 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26232.log 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mgr.x.log: 93.8% -- replaced with /var/log/ceph/objectstore_tool.31979.log.gz 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.34263.log 2024-09-16T04:44:48.707 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26232.log.gz 2024-09-16T04:44:48.708 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26304.log 2024-09-16T04:44:48.713 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/bluestore_tool.34263.log: 93.8% -- replaced with /var/log/ceph/objectstore_tool.31235.log.gz 2024-09-16T04:44:48.713 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32759.log 2024-09-16T04:44:48.713 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26304.log.gz 91.3% -- replaced with /var/log/ceph/bluestore_tool.34263.log.gz 2024-09-16T04:44:48.714 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-16T04:44:48.714 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26528.log 2024-09-16T04:44:48.714 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/objectstore_tool.32759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26480.log 2024-09-16T04:44:48.714 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26528.log.gz 2024-09-16T04:44:48.714 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26280.log 2024-09-16T04:44:48.722 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26480.log.gz 2024-09-16T04:44:48.722 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.33512.log 2024-09-16T04:44:48.722 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26280.log.gz 2024-09-16T04:44:48.722 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26359.log 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/bluestore_tool.33512.log: 86.2% -- replaced with /var/log/ceph/bluestore_tool.33512.log.gz 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26407.log 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26359.log.gz 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26456.log 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26407.log.gz 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26335.log 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26456.log.gz 2024-09-16T04:44:48.723 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26335.log.gz 2024-09-16T04:44:48.731 INFO:teuthology.orchestra.run.smithi165.stderr: 93.7% -- replaced with /var/log/ceph/objectstore_tool.32759.log.gz 2024-09-16T04:44:48.766 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30011.log 2024-09-16T04:44:48.766 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39293.log: /var/log/ceph/ceph-client.admin.33483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33483.log.gz 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31910.log 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31854.log 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30011.log.gz 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31910.log.gz 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39163.log 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37556.log 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31854.log: /var/log/ceph/ceph-client.admin.39163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31854.log.gz 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39163.log.gz 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35799.log 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37144.log 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37556.log.gz 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30012.log 2024-09-16T04:44:48.767 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35799.log.gz 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37144.log.gz 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34297.log 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34734.log 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30012.log: /var/log/ceph/ceph-client.admin.34297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30012.log.gz 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34297.log.gz 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33191.log 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31390.log 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34734.log.gz 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33191.log.gz 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36797.log 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31390.log.gz 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32146.log 2024-09-16T04:44:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37973.log 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36797.log.gz 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32146.log.gz 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37167.log 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31462.log 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37973.log.gz 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37167.log.gz 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32870.log 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31462.log.gz 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29831.log 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38565.log 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32870.log.gz 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29831.log.gz 2024-09-16T04:44:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30912.log 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31463.log 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38565.log.gz 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30912.log.gz 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30824.log 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31463.log.gz 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29984.log 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30824.log.gz 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36194.log 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33080.log 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29984.log.gz 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36194.log.gz 2024-09-16T04:44:48.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37630.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33080.log.gz 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30412.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38669.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37630.log.gz 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30412.log.gz 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33459.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36407.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38669.log.gz 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33459.log.gz 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38084.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36407.log.gz 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34706.log 2024-09-16T04:44:48.771 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38084.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38045.log 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37169.log 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34706.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38045.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33842.log 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38045.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32011.log 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37169.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35609.log 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33842.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32011.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33219.log 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30411.log 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35609.log.gz 2024-09-16T04:44:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33219.log.gz 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34791.log 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35557.log 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30411.log.gz 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39033.log 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34791.log.gz 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35557.log.gz 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32412.log 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33594.log 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39033.log: /var/log/ceph/ceph-client.admin.32412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39033.log.gz 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32412.log.gz 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33114.log 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37129.log 2024-09-16T04:44:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33594.log.gz 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39137.log 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33114.log.gz 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37129.log.gz 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36745.log 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30416.log 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39137.log.gz 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33271.log 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36745.log.gz 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32578.log 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30416.log.gz 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33271.log.gz 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38132.log 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-09-16T04:44:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32578.log.gz 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38132.log.gz 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36875.log 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30430.log 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36875.log.gz 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33110.log 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31208.log 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30430.log.gz 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33110.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.33110.log.gz /var/log/ceph/ceph-client.admin.38513.log 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31208.log.gz 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38539.log 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-09-16T04:44:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38513.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38539.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36615.log 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35635.log 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32691.log 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36615.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35635.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38721.log 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30955.log 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32691.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38721.log.gz 2024-09-16T04:44:48.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35197.log 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37637.log 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30955.log.gz 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35197.log.gz 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35690.log 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33755.log 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37637.log.gz 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35690.log.gz 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36116.log 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31593.log 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33755.log.gz 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36116.log.gz 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38118.log 2024-09-16T04:44:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31593.log.gz 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38118.log.gz 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37135.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30390.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37135.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.38435.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37135.log.gz 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39007.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30390.log.gz 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38435.log.gz 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29988.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38383.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39007.log.gz 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36563.log 2024-09-16T04:44:48.778 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29988.log.gz 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34630.log 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38383.log.gz 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35140.log 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36563.log.gz 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34630.log.gz 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32765.log 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32520.log 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35140.log.gz 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36251.log 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32765.log.gz 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32520.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32520.log.gz 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29876.log 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37564.log 2024-09-16T04:44:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36251.log.gz 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29876.log.gz 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32650.log 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37564.log.gz 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32650.log.gz 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31234.log 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33328.log 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31234.log.gz 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36277.log 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30396.log 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33328.log.gz 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37158.log 2024-09-16T04:44:48.780 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36277.log.gz 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33644.log 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30396.log.gz 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37158.log.gz 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37945.log 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33644.log.gz 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37945.log.gz 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30007.log 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34023.log 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30007.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30007.log.gz -5 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32068.log 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29928.log 2024-09-16T04:44:48.781 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34023.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30395.log 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32068.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26489.log 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29928.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30395.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30938.log 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33785.log 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.26489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26489.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38695.log 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30938.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.33785.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31338.log 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33785.log.gz 2024-09-16T04:44:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38695.log.gz 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33007.log 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35502.log 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31338.log.gz 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33007.log.gz 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30882.log 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30954.log 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35502.log.gz 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30882.log.gzgzip 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33964.log 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30954.log.gz 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30010.log 2024-09-16T04:44:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36771.log 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33964.log.gz 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30010.log.gz 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26250.log 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35057.log 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36771.log.gz 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.26250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26250.log.gz 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32604.log 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33619.log 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35057.log.gz 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32604.log.gz 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37484.log 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36537.log 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33619.log.gz 2024-09-16T04:44:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37484.log.gz 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38825.log 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37540.log 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36537.log.gz 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38825.log.gz 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32386.log 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34598.log 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37540.log.gz 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32386.log.gz 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37675.log 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37997.log 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34598.log.gz 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37675.log.gz 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33553.log 2024-09-16T04:44:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34998.log 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37997.log.gz 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33553.log.gz 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32437.log 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34998.log.gz 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36355.log 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30375.log 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32437.log.gz 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38077.log 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36355.log.gz 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35716.log 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30375.log.gz 2024-09-16T04:44:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38077.log.gz 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39267.log 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34298.log 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35716.log.gz 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39267.log.gz 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34519.log 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38877.log 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34298.log.gz 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34519.log.gz 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36589.log 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32282.log 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38877.log.gz 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36589.log.gz 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37474.log 2024-09-16T04:44:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33509.log 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32282.log.gz 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37474.log.gz 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35900.log 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33509.log.gz 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35900.log.gz 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36927.log 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38929.log 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36927.log.gz 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37156.log 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34401.log 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38929.log.gz 2024-09-16T04:44:48.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36005.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37156.log.gz 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34125.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34401.log.gz 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36005.log.gz 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31654.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34125.log.gz 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38109.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31654.log.gz 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31260.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38109.log.gz 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21462.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38041.log 2024-09-16T04:44:48.789 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31260.log.gz 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.tmp-client.admin.21462.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21462.log.gz 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31364.log 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36041.log 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38041.log.gz/var/log/ceph/ceph-client.admin.31364.log: 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36979.log 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31364.log.gz 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37057.log 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36041.log.gz 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38773.log 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36979.log.gz 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36823.log 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37057.log.gz 2024-09-16T04:44:48.790 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26369.log 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38773.log.gz 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31464.log 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36823.log.gz 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30429.log 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.26369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26369.log.gz 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34816.log 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31464.log.gz 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30429.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30429.log.gz 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36142.log 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36090.log 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34816.log.gz 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36142.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36142.log.gz 2024-09-16T04:44:48.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36090.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.36090.log.gz -- 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.34914.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36667.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.12.log: gzip/var/log/ceph/ceph-client.admin.34914.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38643.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34914.log.gz 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37083.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36667.log.gz 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.38643.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30454.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38643.log.gz 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39422.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37083.log.gz 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30454.log.gz 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37513.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32894.log 2024-09-16T04:44:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39422.log.gz 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38487.log 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37513.log.gz 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29978.log 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32894.log.gz 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30000.log 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38487.log.gz 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30960.log 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29978.log.gz 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31771.log 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30000.log.gz 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30960.log.gz 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34948.log 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30421.log 2024-09-16T04:44:48.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31771.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31420.log 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34948.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34680.log 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30421.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36459.log 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31420.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34680.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29806.log 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32495.log 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36459.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35448.log 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29806.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32495.log.gz 2024-09-16T04:44:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37031.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34208.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35448.log.gz 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30792.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37031.log.gz 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38955.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34208.log.gz 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36485.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30792.log.gz 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30427.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36485.log.gz 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37641.log 2024-09-16T04:44:48.795 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30427.log.gz 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.37641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32844.log 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37641.log.gz 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30802.log/var/log/ceph/ceph-client.admin.38409.log: 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32844.log.gz 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30198.log 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37165.log 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30802.log.gz 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30198.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30418.log 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30198.log.gz 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37165.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37551.log 2024-09-16T04:44:48.796 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37165.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37164.log 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30418.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36303.log 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37551.log: /var/log/ceph/ceph-client.admin.37164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37551.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37164.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36303.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36329.log 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38053.log 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36329.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38053.log.gz 2024-09-16T04:44:48.797 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37005.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30848.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37005.log.gz 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35561.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36901.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30848.log.gz 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35561.log.gz 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29856.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30013.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36901.log.gz/var/log/ceph/ceph-client.admin.29856.log: 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34656.log 2024-09-16T04:44:48.798 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29856.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30013.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30906.log 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35373.log 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34656.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32761.log 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30906.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35373.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.38105.log -- replaced with /var/log/ceph/ceph-client.admin.35373.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30006.log 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32761.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36719.log 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38105.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30006.log.gz 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31743.log 2024-09-16T04:44:48.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32108.log 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36719.log.gz 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34260.log 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31743.log.gz 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36168.log 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32108.log.gz 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34260.log.gz 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34029.log 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39189.log 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36168.log.gz 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35904.log 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34029.log.gz 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38108.log 2024-09-16T04:44:48.800 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39189.log.gz 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35904.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30014.log 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35904.log.gz 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37972.log 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38108.log.gz 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38617.log 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30014.log.gz 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37972.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37972.log.gz -5 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33670.log 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38617.log.gz 2024-09-16T04:44:48.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33670.log.gz 2024-09-16T04:44:48.802 INFO:teuthology.orchestra.run.smithi139.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:44:48.802 INFO:teuthology.orchestra.run.smithi165.stderr: 94.5% -- replaced with /var/log/ceph/objectstore_tool.30491.log.gz 2024-09-16T04:44:48.802 INFO:teuthology.orchestra.run.smithi165.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:44:48.826 INFO:teuthology.orchestra.run.smithi144.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:44:48.984 INFO:teuthology.orchestra.run.smithi059.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T04:44:49.122 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-16T04:44:49.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 has been restored 2024-09-16T04:44:49.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-09-16T04:44:49.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-09-16T04:44:49.353 INFO:teuthology.orchestra.run.smithi144.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T04:44:49.366 INFO:teuthology.orchestra.run.smithi144.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-16T04:44:49.401 INFO:teuthology.orchestra.run.smithi139.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T04:44:49.450 INFO:teuthology.orchestra.run.smithi144.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T04:44:49.503 INFO:teuthology.orchestra.run.smithi165.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T04:44:49.535 INFO:teuthology.orchestra.run.smithi165.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-16T04:44:49.565 INFO:teuthology.orchestra.run.smithi059.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-16T04:44:49.660 INFO:teuthology.orchestra.run.smithi139.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-16T04:44:49.718 INFO:teuthology.orchestra.run.smithi139.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T04:44:49.730 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T04:44:49.739 INFO:teuthology.orchestra.run.smithi165.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-16T04:44:49.793 INFO:teuthology.orchestra.run.smithi165.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T04:44:49.794 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-16T04:44:49.794 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m1.133s 2024-09-16T04:44:49.794 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m3.944s 2024-09-16T04:44:49.795 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.188s 2024-09-16T04:44:49.811 INFO:teuthology.orchestra.run.smithi144.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-16T04:44:49.813 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-16T04:44:49.813 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m1.162s 2024-09-16T04:44:49.813 INFO:teuthology.orchestra.run.smithi144.stderr:user 0m3.306s 2024-09-16T04:44:49.813 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m0.167s 2024-09-16T04:44:50.025 INFO:teuthology.orchestra.run.smithi139.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-16T04:44:50.025 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-09-16T04:44:50.026 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m1.397s 2024-09-16T04:44:50.026 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m4.217s 2024-09-16T04:44:50.026 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m0.147s 2024-09-16T04:44:50.088 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T04:44:50.090 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-16T04:44:50.090 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m1.471s 2024-09-16T04:44:50.090 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m4.152s 2024-09-16T04:44:50.090 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.214s 2024-09-16T04:44:50.091 INFO:tasks.ceph:Archiving logs... 2024-09-16T04:44:50.091 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi059/log 2024-09-16T04:44:50.092 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:44:50.979 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi139/log 2024-09-16T04:44:50.980 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:44:51.603 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi144/log 2024-09-16T04:44:51.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:44:52.091 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi165/log 2024-09-16T04:44:52.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:44:52.688 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T04:44:52.700 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T04:44:52.700 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:44:52.702 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:44:52.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:44:52.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:44:52.819 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T04:44:52.827 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T04:44:52.828 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:44:52.830 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:44:52.833 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:44:52.835 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:44:52.841 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:44:52.841 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2024-09-16T04:44:52.841 INFO:teuthology.orchestra.run.smithi059.stdout:+hv01.front.sepi 67.205.162.81 3 u 16 64 177 0.087 -2.302 1.603 2024-09-16T04:44:52.841 INFO:teuthology.orchestra.run.smithi059.stdout:*hv02.front.sepi 63.231.80.2 3 u 13 64 177 0.083 -1.251 1.439 2024-09-16T04:44:52.842 INFO:teuthology.orchestra.run.smithi059.stdout:+hv03.front.sepi 74.6.168.72 3 u 19 64 177 0.080 +0.663 1.066 2024-09-16T04:44:52.842 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:44:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:44:52.844 INFO:teuthology.orchestra.run.smithi139.stdout:============================================================================== 2024-09-16T04:44:52.844 INFO:teuthology.orchestra.run.smithi139.stdout:*hv01.front.sepi 67.205.162.81 3 u 19 64 177 0.089 -1.498 1.241 2024-09-16T04:44:52.844 INFO:teuthology.orchestra.run.smithi139.stdout:+hv02.front.sepi 63.231.80.2 3 u 20 64 177 0.119 -2.469 0.980 2024-09-16T04:44:52.844 INFO:teuthology.orchestra.run.smithi139.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 64 177 0.073 +1.123 0.737 2024-09-16T04:44:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:44:52.846 INFO:teuthology.orchestra.run.smithi144.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:44:52.846 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================== 2024-09-16T04:44:52.846 INFO:teuthology.orchestra.run.smithi144.stdout:+hv01.front.sepi 67.205.162.81 3 u 16 64 177 0.078 -1.021 1.003 2024-09-16T04:44:52.846 INFO:teuthology.orchestra.run.smithi144.stdout:*hv02.front.sepi 63.231.80.2 3 u 21 64 177 0.111 -1.498 1.438 2024-09-16T04:44:52.846 INFO:teuthology.orchestra.run.smithi144.stdout:+hv03.front.sepi 74.6.168.72 3 u 15 64 177 0.110 +0.938 1.402 2024-09-16T04:44:52.846 INFO:teuthology.orchestra.run.smithi144.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:44:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:44:52.849 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================== 2024-09-16T04:44:52.849 INFO:teuthology.orchestra.run.smithi165.stdout:*hv01.front.sepi 67.205.162.81 3 u 23 64 177 0.082 +0.041 0.704 2024-09-16T04:44:52.849 INFO:teuthology.orchestra.run.smithi165.stdout:+hv02.front.sepi 63.231.80.2 3 u 21 64 177 0.091 -1.135 0.666 2024-09-16T04:44:52.849 INFO:teuthology.orchestra.run.smithi165.stdout:+hv03.front.sepi 74.6.168.72 3 u 18 64 177 0.109 +1.329 0.632 2024-09-16T04:44:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T04:44:52.850 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T04:44:52.859 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T04:44:52.860 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T04:44:52.869 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T04:44:52.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T04:44:52.887 INFO:teuthology.task.internal:Duration was 880.984920 seconds 2024-09-16T04:44:52.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T04:44:52.896 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T04:44:52.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:44:52.899 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:44:52.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:44:52.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:44:52.942 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T04:44:52.942 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-09-16T04:44:52.942 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:44:52.992 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-09-16T04:44:52.992 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:44:53.004 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2024-09-16T04:44:53.004 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:44:53.014 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T04:44:53.015 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:44:53.025 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T04:44:53.025 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:44:53.035 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:44:53.047 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:44:53.058 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:44:53.099 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T04:44:53.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:44:53.107 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:44:53.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:44:53.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:44:53.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T04:44:53.215 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T04:44:53.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:44:53.229 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:44:53.240 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:44:53.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:44:53.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T04:44:53.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.273 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.284 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-09-16T04:44:53.292 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-09-16T04:44:53.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.301 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2024-09-16T04:44:53.314 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-09-16T04:44:53.393 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:44:53.398 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:44:53.403 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:44:53.441 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:44:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:44:53.446 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T04:44:53.456 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T04:44:53.457 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi059 2024-09-16T04:44:53.457 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:44:53.479 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi139 2024-09-16T04:44:53.479 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:44:53.501 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi144 2024-09-16T04:44:53.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:44:53.522 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906025/remote/smithi165 2024-09-16T04:44:53.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:44:53.544 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T04:44:53.544 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:44:53.546 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:44:53.548 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:44:53.567 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:44:53.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T04:44:53.600 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T04:44:53.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T04:44:53.610 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T04:44:53.610 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:44:53.612 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:44:53.615 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:44:53.616 INFO:teuthology.orchestra.run.smithi059.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:44 /home/ubuntu/cephtest 2024-09-16T04:44:53.618 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:44:53.619 INFO:teuthology.orchestra.run.smithi139.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:44 /home/ubuntu/cephtest 2024-09-16T04:44:53.622 INFO:teuthology.orchestra.run.smithi144.stdout: 658459 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:44 /home/ubuntu/cephtest 2024-09-16T04:44:53.637 INFO:teuthology.orchestra.run.smithi165.stdout: 658623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:44 /home/ubuntu/cephtest 2024-09-16T04:44:53.638 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T04:44:53.648 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T04:44:53.674 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 880.9849195480347 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T04:44:53.675 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:44:53.770 INFO:teuthology.run:pass